BC TECH
Saved by @Jacobtldr

We just launched a fun little tool called Phantom Analyzer. It’s a 100% serverless tool that scans websites for hidden tracking pixels.

I want to talk about how we built it 👇

 The idea came about early this year after @mijustin gave us an idea about badges/certification, which developed into “what if we could scan websites for Google Analytics?!”. But we left the idea in Basecamp for many months.
Fast forward to Halloween, we’re thinking about fun ideas we could do to entertain people. We discussed “Phantom Analytics” in response to people getting confused with our product name, and had various ideas. And then we landed on the URL analyzer idea.
Once we’d finalized the spec, Paul got to work on our Halloween themed product and then coded up the HTML/CSS for it all. I then took it, put it into a Laravel application. Nice and easy.
Off the bat, I already knew the base stack I was using.

> Laravel Vapor
> ChipperCI for deployment
> SQS for queues
> DynamoDB for the database

We went with DynamoDB as we don’t want to worry about our database scaling!
So with our infrastructure known, we had a few challenges left to solve:

> How will we scan websites for tracking pixels?
> How will we utilize the queue and check the job is done?
> How will we validate the URL?
For scanning websites, the first thing I did was write out a complex, multi-level regex matching, guzzle executing scanner. But the problem was that it didn’t automatically run the javascript, which often includes additional requests, so the results weren’t accurate.
After spending a long time on that, I was searching for website crawlers and came across Puppeteer, which is a headless Chrome Node.js API. I then searched for how to get it running on Laravel Vapor and saw that someone had already solved that challenge!
I then spent 8 hours trying to start from scratch with Puppeteer, copying from @spatie_be’s Browsershot code, but I just couldn’t get it working. I went to bed, woke up the next morning, and decided that I’d start with Browsershot and simply modify it to what I needed.
I woke up the next day, and within 15 minutes I get a screenshot generated on Laravel Vapor, hooray! I then start to modify Browsershot…

Wait a minute...

Yes, out of the box, Browsershot already had what I needed. Are you kidding me?
So I modified my job and had it all working within minutes. Browsershot returns a list of network requests when loading a web page and returns them, bloody perfect. I then simply compared them against a list of around 10,000 known third party trackers that we had.
The next step was working out how we would get permanent storage in DynamoDB without bringing in anything extra. I wanted to keep it simple. So with DynamoDB as the driver, and “resources” as the storage root, I wrote a command that cached the tracking pixels indefinitely.
One of the initial concerns I had was regarding security. We would be passing user input to the command line and that wouldn't be safe. I spoke with @marcelpociot and he gave me some great advice, and I added in some validation. The active_url rule is fantastic.
I also wanted to have a way so that if a user entered a full URL (e.g. https://t.co/66d4eLDaOu) and not just "https://t.co/GA31muKcta", it still redirected them to the correct results page. Especially since our "tidying up" was opinionated. So we ran this code.
We then had to think about how to configure our Vapor app, and I went with the following settings

> 1024MB of RAM
> 2048 of RAM for the queue (could likely reduce!)
> Warm of 500
> CLI Timeout of 180 seconds

Those settings all worked nicely.
For our Vapor layers, we ran it like this. Very cool. My first time using Layers. Incredible work by the Vapor team (@themsaid @taylorotwell @enunomaduro).
For the "is it ready?" check, I debated using a UUID but I decided that we might have multiple users trying a website at the same time, and they should benefit from the same cache entry (we cache results for 5 minutes).
So for the ping, we went super old school. Interval and redirect when done. Very effective. And when it reloaded the page, it would hit the cache, see the entry and display it.
All in all, this was a fun project to build. I love working with Paul. The only design addition I made was the bats & fade, Paul did everything else. Very grateful for that 😂
I am still in awe over how quickly we deployed this with Vapor. I'm not kidding, it was all coded up and we just created it in the UI, deployed it and we were done. Remarkable experience. Infinite scale without any server work 😎

Hope you all enjoy Phantom Analyzer!

More from Tech

Patrick McKenzie
Patrick McKenzie
@patio11
Torn between "I think losing $100 million when someone beats you at security research is pretty much exactly what you signed up for doing yield farming" and "Maaaaaaaybe not the future of finance you were expecting, huh."

How to put this in regular finance terms...

Suppose hypothetically you have an account at a brokerage with some valuable asset in it. You take a margin loan against that asset to fund your normal spending, or pay a tax bill, or maybe buy something at another brokerage.

For reasons known only to the brokerage, they don't denominate your loan in dollars. They denominate it in shares of a money market fund, which are worth $1 +/- epsilon and basically never deviate from that.

And you think "Hmm, I have a large equity cushion against this loan."

One day, a computer system at the brokerage reports, sorta-kinda erroneously, that the value of the money market fund is actually $1.30 per share. The equity cushion is gone. Your valuable asset is sold, at timing you didn't choose, at wrong price, to pay an inflated phantom debt

And your recourse is... probably tweeting at patio11 saying he finds too much joy in this.

Which I don't; I just feel like this is why you don't trust a CPU built out of redstone to build reliable financial infrastructure on top of.
TECH
Venkatesh Rao
Venkatesh Rao
@vgr
There are only 3 real challenges to free speech at the tech level a) the need for CDNs at scale b) distortions of net neutrality c) nation-state firewalls.

All other claims are about social governance on private sites, where applying free speech doctrine is a stretch at best

Baseline free speech has expanded modestly since 1776 or whatever your reference point is.

About 20% beyond core case of government-critical speech in formally public spaces and media I'd say.

You can (for example) have more kinds of conversations in a bar without getting punched or kicked out by other patrons or management. You can teach more things in colleges than 100 years ago.

That's a genuine de facto increase in practical, informal free speech/expression.

But online free "speech" is, as @reneed has pointed out, not an expansion of free speech at all, but an expansion of reach. It is like the secondary freedom of the press to print as many copies as they can afford, not primary freedom to say what they want

If a newspaper wants to print more copies it has buy more printing equipment and front more upfront costs. Reach for a paper publication is not "free", but a function of desired circulation, which in turn is a function of demand prediction etc. But that logic is obscured online.
TECH
Advertisement
getify
getify
@getify
Java was first released in mid-1995, just 6 months before JS.

By 98, when I went to college, Java was already used for all the first level courses in the CS program.

How did it catch on so quickly (just 3 years) to shift university curriculum, which is usually so slow/behind?

I wonder why Java was so respected and JS was not? Why/how was Java already seen as such a stable and mature language?

Not just Java, other languages (like python) were also released around/near 95, and also seemed to get to "mature" respect status. much quicker.

I bring all this up because my impression of Java in 98 was that it had been around "forever" (as had C++ first developed in 79, but not standardized until 98!), but that's not. true at all. These were all new languages around the same time.

I'm sure the language designs had different merits, and the target applications meant a different perception and reception, but...

it sure seems like the industry (and academics) just decided Java and C++ were the stable mature ones and langs like JS were toys.

BTW, before you assume JS was nascent/immature b/c the web wasn't much of a thing yet, remember that JS was actually released as LiveScript on the server before it was released in a browser. So "server-side JS" was always part of the story, not just Node 15 years later in 2009.
TECH
James Felton
James Felton
@JimMFelton
The Phantom of Heilbronn (or how for 16 years the German police hunted a serial killer who didn't exist)

(Thread, non sweary version here: https://t.co/iDOIhaFgIT)


Between the years of 1993 and 2009, police in Heilbronn, Germany were tracking just about the most elusive serial killer since Ted Cruz.

Known as the Phantom of Heilbronn, or occasionally The Woman Without A Face, she was implicated in so many crimes that the police eventually put out a reward of €300,000 for any information leading to her arrest.

The case that brought her to the media's attention and earned her two ridiculously badass nicknames

(honestly, if you don't want people to do murders you're going to have to come up with less glamorous nicknames - what's wrong with Jack Shitshispants or the Zodiac Cuck)
TECH
Donald @87 \U0001f418
Donald @87 🐘
@disco_donald
@RubinReport Your commentary on the dangers of silicon valley coming under the spell of wokeness and cancel culture is spot on. I know this to be true because in Bitcoin we have already lived through a similar culture war against crypto anarchists.

We experienced firsthand the tactics such groups use to eliminate ¨wrongspreak¨ from public debate. And as you rightly recognize, this exact thing is currently happening in society as a whole. And not surprisingly often by the exact same people.

For example, Jack Dorsey, investor in and honored by the BTC cult, is the same person responsible for the twitter bans of many people, including a lifelong ban for the former president of USA. BTC ideology and such policies may seem incompatible with each other, but it is not.

As we stood up against social engineering and the destruction of language in Bitcoin we have been demonitized and deplatformed. Evidence: Just read the whitepaper and see how BTC, what they tell you is Bitcoin, doesn´t even come close to it.

So, just like you, we have been busy building our own alternative platforms based on new incentive structures and the Own Your Data philosophy.

For example have a look at this excellent presentation on @twetchapp
TECH

You May Also Like

Enigma
Enigma
@ainittomai
On December 25 Mr Pool retweeted August 16 message about the Phoenix and that the worst was still on the horizon to come. 1/*


Maybe now we can assume that November 3 message meant "FUD" nor "FUN". Putting the word in capital letters seems to indicate the aspect the word fud has in our space, FUD. 2/*


Brad started it on December 22 (1/3 posts (Don't say? Sure) = 13 = Plan B). 3/*


And immediately the SEC followed dropping the bomb (1.3 billions (Don't say? Sure) = 13 = Plan B). 4/*


FUD materializes, Bitstamp is one of the first that haltes XRP trading and so on. 5/*
FOR LATER READ
Anu Satheesh
Anu Satheesh
@AnuSatheesh5
Guruvayur Sri Krishna & Coconut with Horns.
#Krishnaleela
#Bhakthi
🕉
We’ve all grown up listening to stories of Shri Krishna. This one is a famous story of Shri Krishna in Guruvayur Temple, also called Dakshina Dwaraka.


There lived a villager who planted coconut saplings and had promised that he would offer the 'first coconut' from each of his coconut trees to "Guruvayurappan". When the trees started to yield coconuts, he collected the first coconut from all the trees in a sack 👇


and set forth to Guruvayur.

On the way he was stopped by a robber who asked all the items in the sack. The villager told the robber that the coconuts in the sack belonged to Guruvayurappan and so he was unable to hand it over.
@SriramKannan77 @Jayalko1


The robber with anger asked the villager "Is Guruvaurappan's coconut any different? Does it have horns ?".
Saying so the robber pulled the sack forcefully out of the villager's hand, and coconuts came out..👇


To their astonishment each and every coconut in the sack had horns! Even today, the coconut with horns are displayed in the temple for devotees to see.

Below Picture is not original, as we cannot take photos inside temple.
Hare Krishna
Guruvayurappa
🕉🕉🕉🕉🕉
Shubham
ALL
Advertisement
raina douris
raina douris
@RahRahRaina
fascinated by this man, mario cortellucci, and his outsized influence on ontario and GTA politics. cortellucci, who lives in vaughan and ran as a far-right candidate for the italian senate back in 2018 - is a major ford donor...


his name might sound familiar because the new cortellucci vaughan hospital at mackenzie health, the one doug ford has been touting lately as a covid-centric facility, is named after him and his family

but his name also pops up in a LOT of other ford projects. for instance - he controls the long term lease on big parts of toronto's portlands... where doug ford once proposed building an nfl stadium and monorail... https://t.co/weOMJ51bVF


cortellucci, who is a developer, also owns a large chunk of the greenbelt. doug ford's desire to develop the greenbelt has been

and late last year he rolled back the mandate of conservation authorities there, prompting the resignations of several members of the greenbelt advisory
WORLD
Sahil
Sahil
@sahilypatel
Want to do an MBA for free ?

A thread 👇

1) Leadership: Foundations of Teamwork and Leadership

School: wharton

2) Marketing: Marketing Management

School: wharton

3) Microeconomics: Microeconomics for Managers

School:

4) Financial Theory

School:
ALL , MBA , MBA SERIES
Professor
Professor
@DillikiBiili
Some ofthe biggest breakouts occur after Volatility Contraction, the analogy is that more a spring is pressed, higher it jumps whenever it gets released.

There is a VCP Scanner on chartink (Not mine). This may also be used to find potential Breakouts.
SCREENERS , TA SCREENERS