portrait picture

TIMO ZIMMERMANN

balancing software engineering & infosec

Automattic & Tumblr

posted on Wednesday 21st of August 2019 in

When it was first announced that Automattic will be acquiring Tumblr I had mixed feelings. Tumblr is still alive, but took so many hits that I was not really sure how much ground it still got to stand on. Plans to merge parts of the architecture with WordPress still seem ambitious to me. And being in a position to argue about and justify the “no adult content policy” surely will not be a pleasant job.

The Verge published an exclusive interview with Matt with many quite interesting points.

Their top priority was not trying to maximize the purchase price. There might even be a corporate reason for the purchase price to be lower, for taxes or something. They were really looking for where the best home was going to be.

Matt @ The Verge

This actually surprises me. Large corporations, including Verizon, usually take a quite different approach at deals. But it is a very pleasant surprise. And considering that Automattic and WordPress, despite all its flaws, is an integral part of the open web, I can see it as a very good home for Tumblr.

The fact that the team can join, a slow transition is planned and a potential independence also speaks for a very thoughtful approach we are usually not seeing in this space. So far this seems like an example of how an acquisition should be dealt with.

Yeah. And some people say, “Well, do you need to be in the app store? Just have a web version.” But apps really are it, and I believe Tumblr is one of the top 30 or 40 apps in the social networking category. It’s usually top couple hundred globally. So their app is a big part of how people interact with it.

Matt @ The Verge

This is totally on point. Without an app Tumblr would lose even more traction. The ease of use, the over all user experience and living on the home screen are essential for an app like Tumblr and for the plans Matt outlines.

And while the AppStore might be the reason why the porn ban was introduced, there should be ways around it. I cannot imagine Apple reviews not being able to find porn on Reddit or Twitter – that would be reason enough to doubt their ability to review anything.

I think there’s a lot of overlap in what both do. I would love for them to interoperate. I do believe that, long-term, there’s an opportunity to merge backend technology so that Tumblr is actually powered by WordPress.

Matt @ The Verge

This surely makes sense from a business and engineering perspective, but please: give me an open source Tumblr frontend for my WordPress setup so I can finally get rid of the garbage I currently have to use to write blog posts. I always liked using the Tumblr interface and the WP web interface just got worse with every single iteration.

I’m getting some strong Google Reader vibes from you. Not that you’re going to build an RSS reader. But it’s still lamented that it’s gone; it was the application that brought together an entire ecosystem of blogs. Is that role something you can fill?

The Verge

It will be interesting to see how ads, content control, integration and monetisation will look like in future. Considering Matts past actions and contributions to the open web there is a chance that we will see a potential competitor for social networks or content publishing emerge that might finally act in users interest and work in an ethical way. I would really appreciate it.

As Matt mentions in the interview it is too soon to see where the journey will lead them, but I am cautiously optimistic that we might get some fresh wind in the online community experience.


Thoughts on video on demand, Netflix and the revival of piracy

posted on Saturday 17th of August 2019 in

Netflix did not really hit their subscriber goal. I actually do not particularly care if the reasons outlined in this article are correct, but I really care about video streaming. And I sadly see the whole thing going downhill so fast that I am waiting for news about „Pirate Bay 2.0 and limewire X“ – our old timers showing up in some new form – bringing back the golden age of piracy.

I am a Netflix subscriber since they expanded to Germany, so somewhen around 2014. 5 years I paid them every single month, the whole family membership, the largest package for the highest quality they can offer. And I am okay with that.

I did not own a TV for nearly a decade and watched all content on my laptop. This worked quite well, until two things happened: Netflix and my wife. So we got a TV. And an AppleTV. All our content on a nice big screen, and I already had a proper speaker system, so this was all setup pretty fast. We started enjoying shows on a 65“ screen instead of a 15“ laptop. Talk about an upgrade. But we still do not have cable or satellite, Netflix and iTunes are our primary sources for movies and TV shows.

My wife sometimes watches the online service of some garbage German TV channel which we are forced to pay for if we want it or not. (It is an astonishingly stupid and people hostile system no one managed to drag through the courts yet…) And last week we bought a movie on Amazon Prime. Not because I like Prime Video – if I would have to pay for it, it would not be available, but hey, it is „free“ – but because iTunes did not have the original dub available.

Recently I started paying for YouTube Premium too, since I am actually watching more YouTube than Netflix and I like some of their shows. Being able to pay instead of being forced to watch ads is exactly the business model people ask for in many online debates around ads, privacy and sustainable business models. I still got a ton of shit for „supporting Google“ and „paying for something I could have for free using $x“. Guess what, this mentality is why we have an ad and privacy-depraved online experience.

For five years all seemed pretty good to me. Then the entertainment industry happened. Everyone wants a piece of the streaming and VOD cake, so Disney started Disney+ for example, with exclusive content. But not only exclusive new content they produce, they also simply remove existing movies and shows from Netflix.

If you look at the larger landscape of upcoming and existing streaming services there is a good chance you will end up in a world where you pay $60-$80 per month, likely even more, to have access to all content.

Obviously people are slightly upset that streaming costs increase by 6-8x per month. We can have a pretty lengthy discussion if the price is actually justified. I subscribe to the theory that the studios and produces can charge whatever they want, if I am willing to pay it is up to me. If I do not consider content worth the price I simply do not consume it. Shows and movies have quite large production costs, so drawing parallels to music does in my opinion not really work out well. In the music industry you also have some alternative income channels like live tours. I mean, I think I would really enjoy seeing Vin Diesel, The Rock and Jason Statham performing a live version of Fast & Furious, but likely „for the lolz“ and not for entertainment value.

But it is not only about cost. With 6-8 streaming sources you will likely be forced to have 6-8 different accounts to manage, 6-8 different applications, one less usable than the other and 6-8 places to search to find the content you are interested in. Apples TV app is trying to streamline this, but as companies still believe in locking users into their shitty experience – hello Twitter – I do not see the TV app being a viable solution in the long run.

With the current developments, there is actually no way forward I see that will serve customers well.

If we have to have multiple subscription based providers a centralized app for consumption would be huge, but is likely not going to happen.

Content being available on all platforms is obviously the best solution, but is likely not going to happen.

A true VOD system where I pay $x to a provider of my choice for the content I want to see, is maybe going to happen. The individual content providers can name their price, but at the same time they do not want a one time payment, but a subscriber.

Remember what happened the last time content was stupidly expensive and inconvenient to access? Piracy. And I can see it happening again considering all the hurdles and money multiple streaming services would bring and cost.

In a small corner of the Internet I can now hear some people scream „ALL CONTENT HAS TO BE FREE“ or „we could pay $10 per month to a central place and they sent $0.01 to content providers which content we consumed“. Take a guess why there are so many shows and movies – because it is a lucrative business. Not going to happen and an absolute stupid idea that is not contributing anything to solving the problem at hand.

At this point I am fairly certain we will actually not be solving the problem, but time will. New platforms with launch, people will subscribe and at some point drop because they do not get enough value for what they pay. At this point prices either will drop or content will be distributed via more channels again. Some platforms like Disney+ have enough money to take a very long time before this happens, but in a few years we will likely only see one or two streaming platforms.

Since this is not a problem I can solve, the only thing left for me to do is figure out what I will be doing in future.

I find myself watching a lot less Netflix lately. New shows which receive a lot of hype simply do not give me a lot. I can watch them, but it is not like I would binge them, a good sign I could live without them. On the other hand I am watching more and more YouTube and Twitch. Especially DIY, gaming and tech channels. To me Netflix is at a point where it is not worth the 15€ per month anymore.

If I really like a show or a movie I usually buy them via iTunes since I will watch them many, many times and I am okay spending a few € for the entertainment value I get out of them. And considering I might have to pay 60-80€ in future, I can buy a few movies and a show per month, not pay anything extra and not hope they are still available when I want to rewatch them.

I do not like the fact that there is DRM in play and that I do not own the media, but that is a trade off I decided to live with for the convenience of having media accessible with one click. Going back to buying things on spinning discs is not really an option for me, there are just too many drawbacks for the impression of ownership.

The entertainment industry is setting themselves up for a stupid battle that will cost them a lot of money and customer satisfaction for some short term profit and a chance at being the winner of the whole streaming and VOD game in a few years. Let us prepare some popcorn and see where this goes, it might be more entertaining than some recent shows.


Zoom, web servers and silent updates

posted on Sunday 14th of July 2019 in ,

Just in case you have not caught the news: Zoom, a video conferencing solution, decided to put a web server on your system which let people remotely dial you into a call, video enabled. Now the ordinary user might think “that’s bad, let me uninstall Zoom”, and they would be right. But there is a catch! Zoom does not uninstall the web server when you uninstall the application. But do not worry, this is not a bug, it is a feature!

After some push back Zoom decided this actually is a big deal and pushed a patch which removes the web server. Well, if you happen to still have Zoom installed, otherwise it will just sit there. Apple decided to jump in and push a silent update – no user interaction required – to remove the web server.

What can we learn from this story?

Well, first of all: Do not use Zoom.

If you are using it, start evaluating alternatives. Not because they messed up, this happens to all. But because they simply disregarded a significant vulnerability, basically told us they do not care about the term “uninstall” doing what a user would expect – hey, it is a feature, right? – and forced third party action to keep users safe. This is simply unacceptable and shows a total lack of understanding when it comes to security.

We all knew that Apple can push silent updates, but I think we will see some discussions around OS ownership once again now that it is in the press. How much control should Apple have over your system without asking or notifying you about it? I am still okay with silent updates – for the vast majority of users this is the exactly the right solution to keep malware and things like Zoom in check. Some day this might change when the first questionable “business decision” is the reason for a silent update, this will be the day the narrative changes – in the end Apple is a business and businesses change leadership, which brings in new and different ideas. I would still hope for a very prominent notification in the form of “$x installed to do $y”.

Going beyond that I would imagine we will see some more similar news in future – I do not believe other video conferencing solutions are actually doing a lot better with all the little helpers, plugins and other crapware they force you to install.

One thing that often comes up in discussions around this incident is what users can do as of today to prevent things like this from happening in future. This is actually a tricky question. I would assume not all people effected by this chose to use Zoom, they were told to by their organisation.

If you have to (or want to) use Zoom or similar: Use a browser without plugins or local applications. Many tools are usable in a modern browser without the need for plugins. Some hide additional features like screen sharing behind a plugin, which many participants in a meeting likely will never need. Prefer plugins over dedicated applications you have to install. I would also suggest evaluating mobile applications – they are often feature complete, do not spin up the fans of your system, work quite well and do not force you to install anything on your workstation.

Lastly: Act. This incident should be enough reason for companies to reevaluate their choice of video conferencing software. Not because they had a vulnerability, again: this happens. But the disregard of it and the blatant ignorance they showed. In a business environment you often discuss critical information – do you really want to trust software and a company that handles security incidents this way?


One week with iPadOS

posted on Friday 5th of July 2019 in , ,

Last week we reached 40°C, or how some of you might call it: 104°F. Sadly the times where you could just go home around 11am because of temperatures approaching something over 30°C are long over. So I still had work to do. But powering on a Xeon system with 5k screens means a lot more heat in my office. So I did the only sensible thing after eyeing the iPadOS beta for some time… I put a beta software on my iPad and started using it exclusively for my daily work. Luckily it was mostly writing specs, reviewing code and doing some work on deploying a new software on an EC2 instance, so all within the realm of iOS capabilities.

While all of the above has been possible with iOS for a long time, iPadOS added so many awesome new features that it made a huge difference during daily usage. While the full set of new features is fairly well documented across the usual news sites and Apples developer page introducing iPadOS, there are a few that really stood out to me using them, especially since I did not consider them to be such a big deal.

Managing my inbox, which has the tendency to significantly fill up over time, became a lot more enjoyable and easy thanks to the new action menu. This seems like a pretty small thing when all you do is hitting reply and write a few lines, but the moment you need to flag, move and organize mails, this actually makes a difference.

The new share sheet also makes life a lot easier. Especially the quick access bar for most recent or relevant contacts – skillfully hidden with all my editing skills – helps a lot when browsing the Internet and finding something you quickly want to share with someone.

Safari is actually one of the biggest improvements, but also still subject to a few inconveniences. You really get a full desktop experience, including a desktop video player for example, which is horrible to use on a touchscreen device. I always preferred the iOS native one over lets say YouTubes bastardisation that did not even support proper fullscreen.

The preview including quick access to downloading, new windows, sharing,… really makes it pleasant to use and a lot easier when you are doing some research for example. The download option is also pretty neat when dealing with PDFs you have to edit anyway or want to read later, at least it feels a lot faster than opening the PDF and saving it to iBooks.

Siri suggestions are actually a lot smarter than I thought. When someone is sending you a link via iMessage it shows up when opening a new tab. This is, again, one of the small things you do not think about a lot, but once you start using it you really appreciate it. The screenshot is, well… I am sure Siri hat a reason for suggesting this link. When there is no special context going on the suggestions sometimes are a bit random.

Being able to resize the keyboard for one handed use is great, with swift typing it is amazing. This is likely the most buggy part of the whole OS right now. Sometimes there are rendering issues, swift typing is as bad as on other devices or third party keyboards when typing German or a mix of German and English and some input elements do not understand the floating keyboard pretty well and are simply hidden behind it – even Apples own ones like iMessage.

Is it ready yet?

As I mentioned I was using the iPad as my only device for a full work week. Media consumption is still amazing on it. Writing and drawing as well. But the rendering bugs can get a bit annoying. I did not have any crashes so far and all third party applications are working. Widgets on your regular home screen are awesome and one click to dial into a hangout or webex call is really nice, if the widgets do not forget to load and display data.

If you rely on less well maintained third party applications – some from banks for example, which constantly seem to cause problems – or want a fully polished experience you should definitely wait.

I did a quick experiment with an external screen connected via HDMI and an USB mouse via AssistiveTouch. It actually worked. It was usable and made the whole thing feel less like an iPad. Mouse input still could use some work as well as an option to use a cursor not designed for people with poor vision. And while I actually like typing on the iPad Smart Keyboard, the moment I had a real screen in front of me my first thought was to pair another Bluetooth keyboard.

All in all I am really happy with the direction of iPadOS and I hope they will iterate a lot more on iPad specific features where it makes sense.


Security 101: Know your threats

posted on Sunday 30th of June 2019 in ,

Engineers often know they have to take security seriously and improve the state their product is in. Non technical management is often worried about security, they know all the horror stories of data leaks and abuse and that this is not always well received by customers. What companies as a whole often do not know is what threats they actually face. Knowing what you try to do is usually a pretty good start.

Let us start our security series with figuring out what threats we should think about in various stages of the startup.

What we will be discussing can be considered threat modeling. Just keep in mind that this will not discuss your full and your specific threat model. We will also not discuss the infamous „nation state attacker“. First of all you will likely not be able to defend against a nation state attack in any way, secondly you would need dedicated professionals to make it a bit harder and thirdly it is likely never a nation state attacker, which is often just a very convenient excuse for “we messed up but want to make it sound like we could do nothing”.

The kid next door

The kid next door is actually a more relevant threat. They usually show up, together with some more serious “bug bounty hunters” or “security researchers” whenever your startup appears on some news portal. They often run an automated or semi automated set of tools like Metasploit, Burp Suite and depending on your marketing page wpscan. The more experienced ones also might play around with your mobile applications or take a look at the source code of your web application to see where assets are hosted and if there is other data they can access by modifying the URL (why you want access controls and no sequential IDs is a topic for another post).

If they find something they consider a vulnerability they likely send you an email asking for the existence of a bug bounty program and if you are willing to pay a few dollars if they disclose their finding. This is not necessarily bad by any means, it gives you an idea of potential attack vectors malicious third parties could actually use against your system.

Running some of those tools yourself is something you can learn. Having the experience to understand the scan results is a bit harder but also doable. But you have to be aware that this is basically a specialization within the whole field of information security, so doing it on the side will always put you at a disadvantage. If you feel like you cannot spend the time on this you should be able to hire an affordable consultant on an ongoing basis who is doing it for you and presents an actionable report.

Engineers

Engineers should appear on ever single list talking about threats. Companies usually trust their engineering department. In the end they are building the software, running the servers and taking care of the data – they surely know what they are doing, right?

In reality they are still humans, do not know every single thing in existence and make mistakes. Sometimes convenience is favored in the decision making process over best practices, other times people simply lack the full understanding of the consequences of certain actions.

Some classics I have seen is the lack of disc encryption while they carry a dump of the production database on their laptop, insufficient access controls and audit controls around production data, misconfiguration of production services and many more things that „should never happen“, but still do.

Engineering is also likely the easiest team to get in line with new security protocols and implementations. There is often the most understanding of the importance of security and more often than you would expect only some small fine tuning of processes and practices is required to get the team to the point you want them to be at.

Your team

Your team, all of your employees, leaders, contractors,…, are basically one big security risk. Again, not the bad one doing things out of bad intent or because they think they should not care, but most likely due to a lack of knowledge or usable solutions for their problems.

Thankfully, same as engineering, people usually understand the importance of security, latest after explaining it to them. Which is the key point, you have to explain to them what to do and more importantly what not to do. Without being arrogant. Or sarcastic. Or believing they are stupid. You know, things you see happening every single day when tech-savvy people with an ego explain something.

You also need to work on tooling. „You cannot share data via DropBox or upload it to Mega“ might be correct – but how are they supposed to share data? The alternative cannot include the words „download gpg“ and „create a key and send it to the key server“. Tools need to be intuitive, easy to use and preferably also look nice.

There will also be people who are upset and actually try to hurt your company in one way or another. Maybe someone gets laid off and is angry – how fast can you lock all their accounts and make sure they do not have access to any company data anymore?

Competitors

Competitors are often very interested in what you are doing, the health of your company and your data. While this is not something that usually happens in the very early stages of a startup, the more funding and traction you gain, the more people will have an interest in those things. Being able to target your customers directly, having an understanding of your finances and investors or simply copying internal parts of your code that differentiate your product are a huge competitive advantage.

Luckily most things we will discuss that prevent accidental data loss also guard against the cheap ways to get to your data – what is left to cover are the more expensive attacks. The moment someone is willing to spend money on getting your data one of the key elements is physical security. How easy is it to get into your network? Are your printers secure? Can people just walk into your office and out with some equipment? Believe me when I say that getting a call starting with „someone walked into the office, put five laptops in a trash bag and walked out“ is not as fun as it sounds years later – thankfully no data was lost this day.

The four horseman of data breaches are a very high level overview of threats you might be facing. Some of them more likely than others, some of them harder to guard against. I hope a bit more understanding of those different kinds of threats will help with the understanding of why some things that seem totally irrelevant are actually necessary and cannot be ignored, no matter in which stage your startup is.

If you want to follow this article series you can either subscribe to the general RSS feed or to the tag specific one if you only care about startup security posts.