BC TECH
Saved by @Jacobtldr

We just launched a fun little tool called Phantom Analyzer. It’s a 100% serverless tool that scans websites for hidden tracking pixels.

I want to talk about how we built it 👇

 The idea came about early this year after @mijustin gave us an idea about badges/certification, which developed into “what if we could scan websites for Google Analytics?!”. But we left the idea in Basecamp for many months.
Fast forward to Halloween, we’re thinking about fun ideas we could do to entertain people. We discussed “Phantom Analytics” in response to people getting confused with our product name, and had various ideas. And then we landed on the URL analyzer idea.
Once we’d finalized the spec, Paul got to work on our Halloween themed product and then coded up the HTML/CSS for it all. I then took it, put it into a Laravel application. Nice and easy.
Off the bat, I already knew the base stack I was using.

> Laravel Vapor
> ChipperCI for deployment
> SQS for queues
> DynamoDB for the database

We went with DynamoDB as we don’t want to worry about our database scaling!
So with our infrastructure known, we had a few challenges left to solve:

> How will we scan websites for tracking pixels?
> How will we utilize the queue and check the job is done?
> How will we validate the URL?
For scanning websites, the first thing I did was write out a complex, multi-level regex matching, guzzle executing scanner. But the problem was that it didn’t automatically run the javascript, which often includes additional requests, so the results weren’t accurate.
After spending a long time on that, I was searching for website crawlers and came across Puppeteer, which is a headless Chrome Node.js API. I then searched for how to get it running on Laravel Vapor and saw that someone had already solved that challenge!
I then spent 8 hours trying to start from scratch with Puppeteer, copying from @spatie_be’s Browsershot code, but I just couldn’t get it working. I went to bed, woke up the next morning, and decided that I’d start with Browsershot and simply modify it to what I needed.
I woke up the next day, and within 15 minutes I get a screenshot generated on Laravel Vapor, hooray! I then start to modify Browsershot…

Wait a minute...

Yes, out of the box, Browsershot already had what I needed. Are you kidding me?
So I modified my job and had it all working within minutes. Browsershot returns a list of network requests when loading a web page and returns them, bloody perfect. I then simply compared them against a list of around 10,000 known third party trackers that we had.
The next step was working out how we would get permanent storage in DynamoDB without bringing in anything extra. I wanted to keep it simple. So with DynamoDB as the driver, and “resources” as the storage root, I wrote a command that cached the tracking pixels indefinitely.
One of the initial concerns I had was regarding security. We would be passing user input to the command line and that wouldn't be safe. I spoke with @marcelpociot and he gave me some great advice, and I added in some validation. The active_url rule is fantastic.
I also wanted to have a way so that if a user entered a full URL (e.g. https://t.co/66d4eLDaOu) and not just "https://t.co/GA31muKcta", it still redirected them to the correct results page. Especially since our "tidying up" was opinionated. So we ran this code.
We then had to think about how to configure our Vapor app, and I went with the following settings

> 1024MB of RAM
> 2048 of RAM for the queue (could likely reduce!)
> Warm of 500
> CLI Timeout of 180 seconds

Those settings all worked nicely.
For our Vapor layers, we ran it like this. Very cool. My first time using Layers. Incredible work by the Vapor team (@themsaid @taylorotwell @enunomaduro).
For the "is it ready?" check, I debated using a UUID but I decided that we might have multiple users trying a website at the same time, and they should benefit from the same cache entry (we cache results for 5 minutes).
So for the ping, we went super old school. Interval and redirect when done. Very effective. And when it reloaded the page, it would hit the cache, see the entry and display it.
All in all, this was a fun project to build. I love working with Paul. The only design addition I made was the bats & fade, Paul did everything else. Very grateful for that 😂
I am still in awe over how quickly we deployed this with Vapor. I'm not kidding, it was all coded up and we just created it in the UI, deployed it and we were done. Remarkable experience. Infinite scale without any server work 😎

Hope you all enjoy Phantom Analyzer!

More from Tech

Yoel Roth
Yoel Roth
@yoyoel
We’ve recently seen research about so-called “bots” and misinformation on Twitter and wanted to share our perspective on why findings that might seem remarkable at first are likely inaccurate. We’re working on a more detailed explanation, but some comments for now.

We continue to be excited by the research opportunities that Twitter data provides. Our service is the largest source of real-time social media data, and we make this data available to the public for free through our public API. No other major service does this.

Many researchers, academics, and journalists use our public API — a set of tools for programmatically accessing information on Twitter. We make all public Twitter content available via our APIs. You can learn more about them here:

The basic issue with much of the research based on our public APIs is simple: The APIs don't provide insight into our defensive actions to protect Twitter from manipulation, including bots.

Because of this, API-based research can't distinguish between accounts we've already identified as bad (and hidden or removed) and real, authentic ones.
TECH
Guillaume Chaslot
Guillaume Chaslot
@gchaslot
The YouTube algorithm that I helped build in 2011 still recommends the flat earth theory by the *hundreds of millions*. This investigation by @RawStory shows some of the real-life consequences of this badly designed AI.


This spring at SxSW, @SusanWojcicki promised "Wikipedia snippets" on debated videos. But they didn't put them on flat earth videos, and instead @YouTube is promoting merchandising such as "NASA lies - Never Trust a Snake". 2/


A few example of flat earth videos that were promoted by YouTube #today:
https://t.co/TumQiX2tlj 3/

https://t.co/uAORIJ5BYX 4/

https://t.co/yOGZ0pLfHG 5/
TECH , FLAT EARTH
Advertisement
Wilbert Liu \U0001f468\U0001f3fb\u200d\U0001f3a8
Wilbert Liu 👨🏻‍🎨...
@wilbertliu
These past few days I've been experimenting with something new that I want to use by myself.

Interestingly, this thread below has been written by that.

Let me show you how it looks like. 👇🏻


When you see localhost up there, you should know that it's truly an experiment! 😀


It's a dead-simple thread writer that will post a series of tweets a.k.a tweetstorm. ⚡️

I've been personally wanting it myself since few months ago, but neglected it intentionally to make sure it's something that I genuinely need.

So why is that important for me? 🙂

I've been a believer of a story. I tell stories all the time, whether it's in the real world or online like this. Our society has moved by that.

If you're interested by stories that move us, read Sapiens!

One of the stories that I've told was from the launch of Poster.

It's been launched multiple times this year, and Twitter has been my go-to place to tell the world about that.

Here comes my frustration.. 😤
TECH
Taylor Holiday
Taylor Holiday
@TaylorHoliday
1/ What causes long term sustained performance in an ad account?

🧵featuring real examples 👇🏻

If you explore the source of FB ad growth and subsequently revenue growth for eCom brands it is almost never a function of iterative improvements to ad creative or tactics over time.

2/ More often it is a series of moments that unlock an order of magnitude increase in awareness, engagement, traffic and performance.

These moments can be caused by:

A single ad
A big PR moment
A breakthrough campaign
A change in market dynamics
A new product release
Etc.

3/ The problem is they are very hard to predict and create.

I’ll give you an example of a few that happened for us @QALORing that changed our trajectory (and ad account performance) each time...

4/ The first was @andydalton14 wearing a ring on @HBO hard knocks. It was the peak of the series popularity and they did a :30 piece on the ring he was wearing while playing.

See screenshot

Notice the traffic before and after the peak.


5/ This increased organic traffic let us alter our AdWords strategy, extract more post intent demand from new searchers and pull clips of the show to supplement our ad credibility.
TECH
Lea Kissner
Lea Kissner
@LeaKissner
Next up in Privacy Technology at #enigma2021, Kelly Huang from @ethyca speaking about "GONE, BUT NOT "FORGOTTEN"—TECHNICAL & PRACTICAL CHALLENGES IN OPERATIONALIZING MODERN PRIVACY

Just imagine there's a global pandemic forcing everyone to stay home and buy their stuff over the internet. And you've been working on your sanitization-on-demand startup. You've got more users than you can count! ... literally, because your data's all over.

Now you're a multi-national international country with privacy issues because your information is all over the place.

Now a user writes to request you delete their data. Where is it? How do you do that? Who's responsible for privacy in your business.

How do you operationalize privacy rights?

Primary stakeholders:
* Legal
* Business
* Engineering

We spend a lot of time on Twitter analyzing the legal rulings, but it's harder where the "rubber meets the code" 🥁
TECH

You May Also Like

Jaya_Upadhyaya
Jaya_Upadhyaya
@Jayalko1
SRI SWETHA VINAYAGAR, THIRUVALANCHUZHI, KUMBAKONAM
(TN)
#bharatmandir

While main deity is Shiva as Kapartheeswarar, the kshetram is famous for Ganesha, known as Swetha/Vellai Vinayagar (white Vinayak)

During samudra manthan, Vasuki snake (used as rope) started spewing venom.


Unable to bear the intense heat, Devas rushed to Siva. He told them that this was because the task was started without getting Ganesha’s approval.
So the Devas created a Ganesha murti from white sea foam and worshipped it.
It was later taken by Indra to Indralok.
@GunduHuDuGa


But once Indra had to come to Earth to pray to Shiva to seek relief from a curse inflicted on him. So he brought the murti of Ganesha with him. When he came here, he wanted to take bath before puja. So he gave the murti to a boy to hold for sometime. The boy was actually Shiva.


He kept the murti down and vanished. Indra could not lift it again and a divine voice told him to let Vinayagar stay there and visit every Ganesh Chaturthi to pray.
Since Vinayagar is made of foam, no abhishekam is done here. Only punugu is applied n camphor powder is sprinkled
ALL
Jig's Patel
Jig's Patel
@jigspatel1988
What is best way to sell strangle in Banknifty on an intraday basis
Simple
at 9:45 Open Option chain and check In which strike highest OI change in call side and put side

Sell both option with combine SL
Enjoy easy and simple intraday trading strategy
Today 35000CE and 34500PE
OPTIONSLEARNINGS , OPTION LEARNING , OPTIONSELLING
Advertisement
Simon DeDeo
Simon DeDeo
@SimonDeDeo
This is a pretty valiant attempt to defend the "Feminist Glaciology" article, which says conventional wisdom is wrong, and this is a solid piece of scholarship. I'll beg to differ, because I think Jeffery, here, is confusing scholarship with "saying things that seem right".


The article is, at heart, deeply weird, even essentialist. Here, for example, is the claim that proposing climate engineering is a "man" thing. Also a "man" thing: attempting to get distance from a topic, approaching it in a disinterested fashion.


Also a "man" thing—physical courage. (I guess, not quite: physical courage "co-constitutes" masculinist glaciology along with nationalism and colonialism.)


There's criticism of a New York Times article that talks about glaciology adventures, which makes a similar point.


At the heart of this chunk is the claim that glaciology excludes women because of a narrative of scientific objectivity and physical adventure. This is a strong claim! It's not enough to say, hey, sure, sounds good. Is it true?
ALL
Noah Smith
Noah Smith
@Noahpinion
1/People I know on the Right tend to be obsessed with the idea of "crimethink", taboos, and the (supposedly) oppressive, omnipresent enforcement of liberal cultural norms.

Why?

My new theory: A lot of it is guilt.

2/It's true that in some circles - the media, universities, many big corporations - liberal norms *are* enforced to some degree, and the enforcement has probably gotten stronger in recent years.

But I don't think this explains all of the Right's obsession.

3/The stereotype is that people on the Right tend to be more threat-sensitive. That would help explain the perception that liberal norm-enforcers are lurking everywhere, ready to pounce and anathematize anyone who makes a misstep.

But I don't think that's all of it either.

4/And of course, it's always fun - whether you're on the Right or the Left, or playing a video game, or whatever - to style yourself as the brave iconoclastic rebel fighting against the oppressive forces of blah blah blah.

But I don't think this is all of it, either.

5/The reason I think it's more than these things is that even *in private*, people on the Right express their "crimethink" ideas very gingerly and hesitantly.

I get the sense that they're not just afraid of external censure, but have also internalized liberal norms.
SOCIETY
XVIII Airborne Corps\U0001f409
XVIII Airborne Corps🐉...
@18airbornecorps
1 of 16: WE ARE ALL JEWS HERE: THE STORY OF RODDIE EDMONDS

One of the most moving and relevant stories of the Battle of the Bulge, or any American Soldier in any war, is that of Master Sergeant Roddie Edmonds, a Knoxville, Tennessee native, who served with the 106th Infantry.


2 of 16:

Roddie was captured early on in the Battle of the Bulge, on December 19th, when Panzer forces plowed through his unit.

He, along with almost his entire regiment, was forced to surrender.


3 of 16:

The men were transported to the Stalag IX-A POW camp in Ziegenhain, Germany.

Roddie was the senior enlisted American Soldier at the site. As such, he was the conduit between all American Soldiers and their German captors.


4 of 16:

In late January, the camp’s commandant, Major Siegmann [pictured here], ordered Roddie, a Christian, to identify all Jewish Soldiers and order them to stand in formation by themselves the next day.

[Jewish Soldiers were a minority within American units]


5 of 16:

Throughout WWII, captured Jewish Soldiers were often tortured and then killed by their German captors.

In fact, Jewish Soldiers had been told to bury their dog tags before capture. The dog tag identified Jews with the letter “H” for Hebrew.
SOCIETY