We just launched a fun little tool called Phantom Analyzer. It’s a 100% serverless tool that scans websites for hidden tracking pixels.

I want to talk about how we built it 👇

The idea came about early this year after @mijustin gave us an idea about badges/certification, which developed into “what if we could scan websites for Google Analytics?!”. But we left the idea in Basecamp for many months.
Fast forward to Halloween, we’re thinking about fun ideas we could do to entertain people. We discussed “Phantom Analytics” in response to people getting confused with our product name, and had various ideas. And then we landed on the URL analyzer idea.
Once we’d finalized the spec, Paul got to work on our Halloween themed product and then coded up the HTML/CSS for it all. I then took it, put it into a Laravel application. Nice and easy.
Off the bat, I already knew the base stack I was using.

> Laravel Vapor
> ChipperCI for deployment
> SQS for queues
> DynamoDB for the database

We went with DynamoDB as we don’t want to worry about our database scaling!
So with our infrastructure known, we had a few challenges left to solve:

> How will we scan websites for tracking pixels?
> How will we utilize the queue and check the job is done?
> How will we validate the URL?
For scanning websites, the first thing I did was write out a complex, multi-level regex matching, guzzle executing scanner. But the problem was that it didn’t automatically run the javascript, which often includes additional requests, so the results weren’t accurate.
After spending a long time on that, I was searching for website crawlers and came across Puppeteer, which is a headless Chrome Node.js API. I then searched for how to get it running on Laravel Vapor and saw that someone had already solved that challenge!
I then spent 8 hours trying to start from scratch with Puppeteer, copying from @spatie_be’s Browsershot code, but I just couldn’t get it working. I went to bed, woke up the next morning, and decided that I’d start with Browsershot and simply modify it to what I needed.
I woke up the next day, and within 15 minutes I get a screenshot generated on Laravel Vapor, hooray! I then start to modify Browsershot…

Wait a minute...

Yes, out of the box, Browsershot already had what I needed. Are you kidding me?
So I modified my job and had it all working within minutes. Browsershot returns a list of network requests when loading a web page and returns them, bloody perfect. I then simply compared them against a list of around 10,000 known third party trackers that we had.
The next step was working out how we would get permanent storage in DynamoDB without bringing in anything extra. I wanted to keep it simple. So with DynamoDB as the driver, and “resources” as the storage root, I wrote a command that cached the tracking pixels indefinitely.
One of the initial concerns I had was regarding security. We would be passing user input to the command line and that wouldn't be safe. I spoke with @marcelpociot and he gave me some great advice, and I added in some validation. The active_url rule is fantastic.
I also wanted to have a way so that if a user entered a full URL (e.g. https://t.co/66d4eLDaOu) and not just "https://t.co/GA31muKcta", it still redirected them to the correct results page. Especially since our "tidying up" was opinionated. So we ran this code.
We then had to think about how to configure our Vapor app, and I went with the following settings

> 1024MB of RAM
> 2048 of RAM for the queue (could likely reduce!)
> Warm of 500
> CLI Timeout of 180 seconds

Those settings all worked nicely.
For our Vapor layers, we ran it like this. Very cool. My first time using Layers. Incredible work by the Vapor team (@themsaid @taylorotwell @enunomaduro).
For the "is it ready?" check, I debated using a UUID but I decided that we might have multiple users trying a website at the same time, and they should benefit from the same cache entry (we cache results for 5 minutes).
So for the ping, we went super old school. Interval and redirect when done. Very effective. And when it reloaded the page, it would hit the cache, see the entry and display it.
All in all, this was a fun project to build. I love working with Paul. The only design addition I made was the bats & fade, Paul did everything else. Very grateful for that 😂
I am still in awe over how quickly we deployed this with Vapor. I'm not kidding, it was all coded up and we just created it in the UI, deployed it and we were done. Remarkable experience. Infinite scale without any server work 😎

Hope you all enjoy Phantom Analyzer!

More from Tech

"I really want to break into Product Management"

make products.

"If only someone would tell me how I can get a startup to notice me."

Make Products.

"I guess it's impossible and I'll never break into the industry."

MAKE PRODUCTS.

Courtesy of @edbrisson's wonderful thread on breaking into comics –
https://t.co/TgNblNSCBj – here is why the same applies to Product Management, too.


There is no better way of learning the craft of product, or proving your potential to employers, than just doing it.

You do not need anybody's permission. We don't have diplomas, nor doctorates. We can barely agree on a single standard of what a Product Manager is supposed to do.

But – there is at least one blindingly obvious industry consensus – a Product Manager makes Products.

And they don't need to be kept at the exact right temperature, given endless resource, or carefully protected in order to do this.

They find their own way.
Thought I'd put a thread together of some resources & people I consider really valuable & insightful for anyone considering or just starting out on their @SorareHQ journey. It's by no means comprehensive, this community is super helpful so no offence to anyone I've missed off...

1) Get yourself on the official Sorare Discord group
https://t.co/1CWeyglJhu, the forum is always full of interesting debate. Got a question? Put it on the relevant thread & it's usually answered in minutes. This is also a great place to engage directly with the @SorareHQ team.

2) Bury your head in @HGLeitch's @SorareData & get to grips with all the collated information you have to hand FOR FREE! IMO it's vital for price-checking, scouting & S05 team building plus they are hosts to the forward thinking SO11 and SorareData Cups 🏆

3) Get on YouTube 📺, subscribe to @Qu_Tang_Clan's channel https://t.co/1ZxMsQR1kq & engross yourself in hours of Sorare tutorials & videos. There's a good crowd that log in to the live Gameweek shows where you get to see Quinny scratching his head/ beard over team selection.

4) Make sure to follow & give a listen to the @Sorare_Podcast on the streaming service of your choice 🔊, weekly shows are always insightful with great guests. Worth listening to the old episodes too as there's loads of information you'll take from them.

You May Also Like

First update to https://t.co/lDdqjtKTZL since the challenge ended – Medium links!! Go add your Medium profile now 👀📝 (thanks @diannamallen for the suggestion 😁)


Just added Telegram links to
https://t.co/lDdqjtKTZL too! Now you can provide a nice easy way for people to message you :)


Less than 1 hour since I started adding stuff to https://t.co/lDdqjtKTZL again, and profile pages are now responsive!!! 🥳 Check it out -> https://t.co/fVkEL4fu0L


Accounts page is now also responsive!! 📱✨


💪 I managed to make the whole site responsive in about an hour. On my roadmap I had it down as 4-5 hours!!! 🤘🤠🤘
Recently, the @CNIL issued a decision regarding the GDPR compliance of an unknown French adtech company named "Vectaury". It may seem like small fry, but the decision has potential wide-ranging impacts for Google, the IAB framework, and today's adtech. It's thread time! 👇

It's all in French, but if you're up for it you can read:
• Their blog post (lacks the most interesting details):
https://t.co/PHkDcOT1hy
• Their high-level legal decision: https://t.co/hwpiEvjodt
• The full notification: https://t.co/QQB7rfynha

I've read it so you needn't!

Vectaury was collecting geolocation data in order to create profiles (eg. people who often go to this or that type of shop) so as to power ad targeting. They operate through embedded SDKs and ad bidding, making them invisible to users.

The @CNIL notes that profiling based off of geolocation presents particular risks since it reveals people's movements and habits. As risky, the processing requires consent — this will be the heart of their assessment.

Interesting point: they justify the decision in part because of how many people COULD be targeted in this way (rather than how many have — though they note that too). Because it's on a phone, and many have phones, it is considered large-scale processing no matter what.
The chorus of this song uses the shlokas taken from Sundarkand of Ramayana.

It is a series of Sanskrit shlokas recited by Jambavant to Hanuman to remind Him of his true potential.

1. धीवर प्रसार शौर्य भरा: The brave persevering one, your bravery is taking you forward.


2. उतसारा स्थिरा घम्भीरा: The one who is leaping higher and higher, who is firm and stable and seriously determined.

3. ुग्रामा असामा शौर्या भावा: He is strong, and without an equal in the ability/mentality to fight

4. रौद्रमा नवा भीतिर्मा: His anger will cause new fears in his foes.

5.विजिटरीपुरु धीरधारा, कलोथरा शिखरा कठोरा: This is a complex expression seen only in Indic language poetry. The poet is stating that Shivudu is experiencing the intensity of climbing a tough peak, and likening

it to the feeling in a hard battle, when you see your enemy defeated, and blood flowing like a rivulet. This is classical Veera rasa.

6.कुलकु थारथिलीथा गम्भीरा, जाया विराट वीरा: His rough body itself is like a sharp weapon (because he is determined to win). Hail this complete

hero of the world.

7.विलयगागनथाला भिकारा, गरज्जद्धरा गारा: The hero is destructive in the air/sky as well (because he can leap at an enemy from a great height). He can defeat the enemy (simply) with his fearsome roar of war.