portrait picture

TIMO ZIMMERMANN

balancing software engineering & infosec

Security 101: Path of least resistance

posted on Thursday 28th of May 2020 in ,

No matter if you are starting your career in information security, want to build your newly started company on a solid foundation or for whatever reason decided that it is time to step up your security and compliance game: You are in for a long journey with dozens of fights and surprises you will never anticipate. In the spirit of covering as many angles of possible related to security, let us talk a bit about you and your colleagues.

Let us start with a straight forward example. You decided to start a company working in a highly regulated field, like healthcare. If you are based in the US this most likely means you have to be HIPAA compliant. Most founders are aware of this requirement and have a rough understanding of the implications, but it is likely not deep and not technical enough to ensure that a growing and complex web or mobile app satisfies all requirements. This is why people specialized in security and compliance – like yourself – are brought in.

Most companies will have someone running marketing. No matter how good your product is, you have to sell it. And all over the world one thing holds true to marketing: they need data. The more data the better. This is their job. And let me tell you, it is not very compatible with highly regulated fields. So their first request to engineering is putting Google Analytics on the website so they have an idea what visitors look at and when they lose interest.

As shocking as this might sound, this is actually a problem. Google is actually very specific to not simply use GA for a HIPAA compliant product – believe me, no one will every read this FAQ when tasked to add GA. There is a good chance that you will hear some form of justification along the lines of „but look at healthCareCompanyX! They also use GA“ when you raise this issue. The sad truth is that there are tons of compliance violations in every single field by companies of any size. As long as no one catches it during an audit or sues it simply will not change. This does not mean it is okay to join the ranks of companies violating regulations and therefore customer trust.

There are a few different actions I have seen security teams take in response to a compliance violation like the one described.

You can simply remove GA from the app or website. Now your marketing team does not have data anymore they actually need to be efficient at their job. You did not just take out a whole teams productivity, but most likely also have your leadership team question why they pay a few people a lot of money to sit around. This might sound a little bit dramatic, but it is one of the outcomes I witnessed.

It does not even matter if there is any truth to it, you are in a bad position at this point. Compliance will win lots of arguments and drive decisions. But the team will remember that you turned off one of the tools they need. And moving forward they will try to make sure you do not know about the tools they use. You are the villain that tries to prevent them from doing their job!

Sooner or later you will find mysterious services and tools you never knew your company is using – and chances are pretty good they will not be compliant with your regulations. People will try to take the path of least resistance – which is not talking to you and simply start using tools. When it is uncovered there are usually no ramifications for them, in the end they “did what they had to do to get their job done”.

Another option is to replace GA. You do some research and notice that you can self-host Matomo in a HIPAA compliant way. And it shows some fancy graphs that look like GAs, so it has to be a good replacement, right? Set up a server, drop the pixel and you are done. Except you forgot to port data from GA over. Existing data is core to an analytics solution, you cannot simply discard months of data. Except the marketing team does not know how to use the new tool. Except that core features work differently. Except… you see where this is going.

Professionals usually know one or two tools. They invest a lot to get really good at using those tools to get their job done. Sometimes there is only one tool the whole industry seems to use. You cannot „simply start using another one“. You might have to get a consultant on board for custom development and training. You need to plan transitioning periods. And all of this assumes the team understands why you want to force them to abandon the tool that served them so well for all of their career.

The least fruitful „solutions“ – let us be charitable and call them that – I have ever seen was creating a ticket for the marketing team telling them „GA is not HIPAA compliant – please find another tool“ or asking large enterprises who are not setup for it to sign a BAA. Good luck, you need it.

On a personal note: I have chosen this example because it is easy to follow, this is not a general statement about marketing teams. The same scenario will happen in every team across your company – no matter if it is marketing, finance or engineering. I have also seen amazing teams being sensitive to compliance requirements and doing all the work upfront to make sure tooling will be compliant.

I have never witnessed a good security team that worked in isolation behind closed doors. You can do all the research you want and decide to move forward with what the industry considers best practices and top of the line security measures, but if your team and company is not on board you basically burn money and set yourself up for failure.

The two most important things you have to work on as early as possible are making sure everyone in your company is aligned on the importance of security and compliance and being their partner figuring out solutions that work from a security and compliance point of view – as well as setting your colleagues up for success.

Understanding the importance of compliance and the basic nuances of how decisions are made and tools are evaluated will help teams a lot when making the initial request. In a perfect world you provide something like a small check list to them and as they engage a new vendor to simply check off boxes so they can have a higher confidence that the work they put in talking to them will be worth their time.

Or, if some of the boxes are not checked they know they will have to pull in you and your team immediately before spending more time on the partnership. Self service is one of the most important aspects, as it allows teams to move as fast as they want, not as fast as you are able to do your research. Security teams usually do not scale as required all the time. And the moment it becomes bothersome, slow or tedious to work with you, people will again follow the principle of least effort which is cutting you out of the loop, delaying as long as possible to bring you in.

At some point any team that wants a new, shiny tool should talk to you and your team for a final sign off. The sooner in the process the better. And best case you are involved in the vendor selection early on and act as a thought partner. This will not always be feasible. But having teams follow some form of sign off process and making sure they know you work with them and try to make things work is far better than working reactionary to signed deals. Once you committed money you often will be told to make it work. Even if it is not possible, you will be asked to get creative. This is even less fun than it sounds.

Security and compliance is a process involving the whole company. Work with people and set them up for success. Be mindful of their time and requirements. Help them to get to a solution that works for them, not only for you. This will make your job so much easier and more pleasant.

If you want to follow this article series you can either subscribe to the main RSS feed or to the tag specific one if you only care about startup security posts.


Security 101: Identity Providers

posted on Tuesday 26th of May 2020 in ,

There are only a few things I am aware of that most startups care less about than employee management, on and off boarding and access management. Especially when just starting your journey with two friends it sounds like something totally unnecessary… and then you hire your first few employees. Obviously you conduct – what you believe to be – a job interview, and you are obviously sure you hired the absolute right people. Why add tons of additional work? You can trust them. You hired them. They are „good people“. Right?

Wrong.

You might actually be right, but this will be an outlier as you grow your company. You sometimes will hire the wrong people. Maybe they are not as productive as you expect. Maybe they do not play well with your existing org structure. Or they struggle with adjusting to the size of your team. Maybe they turn out to be assholes. Fact is, at some point you will have to let people go. And if you do it for the first time – maybe for the first 50 times – chances are pretty high you will make mistakes. Chances are also pretty high people you let go will be displeased, at least.

What is the worst that can happen? They might delete your production infrastructure. They might push malicious commits to your repository. They might dump your database and sell it to a direct competitor. If you believe this is far fetched you are way off. All of the above happens, more often than you believe and more often than anyone of us likes.

The inevitable truth is that as your organization grows people you have hired will do things that will cause harm to your company – out of spite, not knowing better or simply an accident. From my experience this is one of the toughest lessons to communicate to a founder or small team. Something we do not like, because it means admitting we made a mistake hiring someone, overestimated their capabilities or did not foresee that we are all humans and make mistakes on a regular basis.

One of the most fundamental parts of your software environment will be an identity provider. An IdP usually has at least two responsibilities:

  1. manage user accounts for all your employees
  2. provide a way for other systems to authenticate and authorize users

You create exactly one user account when you hire an employee. You assign them one or multiple roles and once they set a password and complete the two factor authentication setup they are granted access to all relevant systems. You part with an employee and you deactivate the account in your identity provider, blocking access to all of your company systems. This is obviously the very simplified description of an IdP and most services and systems provide additional functionality (like authenticating system users on employees laptops or integrating with your network gear).

Picking an IdP is a bigger deal than choosing an analytics solution you can swap for a new one by replacing a pixel or a few lines of JavaScript. Your IdP on the other hand will be – ideally – tied to every single system you use. Every single one. Switching providers is expensive and error prone. Which is one of the reasons why the sales teams of most companies operating in this sector are far better than their support.

So how to best approach this problem? I would suggest incrementally. Most software you will be using early on should be able to leverage Google for SSO. Paired with G Suite, which most startups will likely be using anyway, you have a solution in place that checks most of the boxes.

This is one of the points where I support pragmatism over best practices. You will not be able to reap all the benefits of an IdP with Google. Actually far from it. But you are able to easily on and off board employees and enforce some security best practices. I consider this a win. And it gets out of your three person teams way, so setup and maintenance are not an excuse to fall back to sharing a single account via Slack among all people.

There will be a few apps that will not support Single Sign On (SSO). In this case there are two options. First: use something else. Second: talk to them and see if they can provide SSO. If both options are not viable you will have to fall back to creating user accounts. Plural, please. Add an explicit action item to your off boarding checklist to deactivate or delete those accounts. Also – create the checklist.

As your team grows you want to move to a more advanced solution like JumpCloud or Okta. But this will come with additional maintenance requirements. You will need to understand some concepts like SAML and find your way through interfaces not adhering to any standard but assuming you already know exactly what you are doing when you upload your newly generated SSL certificates. Most of the time SAML also comes with an enterprise plan you have to pay for. Only enterprise users care about security and an IdP, right? And enterprise users always have a lot of money to burn, right? SaaS vendors artificially gate security features, and from a vendor perspective this makes a lot of sense. For you it sucks.

Once you adopt an IdP I would suggest going all in: Let it manage your SSH keys to ssh into your servers. Use it for system logins. Use their Radius server to authenticate users connecting to your WiFi. Go through their feature list and try to use them to its fullest. The less manual steps you have to take and the faster you can deactivate accounts – including in the event of one being compromised – the better.

As I said earlier: This is some serious work to get it all right and rolled out. But it is one of the things that actually moves the needle on the security meter quite a bit.

If you want to follow this article series you can either subscribe to the main RSS feed or to the tag specific one if you only care about startup security posts.


Python – convert JSON to CSV

posted on Thursday 21st of May 2020 in ,

Assume you have an API which returns some data. You want to provide the data to someone who wants to load it in Excel and do some analysis. Yes, this actually still happens. Yes, this is fine, it works and gets people what they need. But writing a response transformer for your API to spit out a proper CSV is a bit tedious if your framework does not provide this functionality. Let us fix this in the most easy way which is also a total overkill considering the tool we will use.

Your API needs to return consistent data. Something like [{"x": "1"}, {"y": "2"}, 3, 4,] will not work. The good news is most (pseudo) REST like JSON APIs will work. First we dump the response data to a file.

Now to convert the response data to a CSV. We will use Pandas. I told you the solution will be a total overkill.

import pandas
dfm = pandas.read_json("path_to_json_file.json")
dfm.to_csv("win.csv")

That is it.

Just in case you want to add CSV output to your app you can also do this in memory.

import pandas
import requests

response = requests.get("https://foo.tld/api/…"))

csv = io.StringIO()

dfm = pandas.DataFrame.from_records(response.json())
dfm.to_csv(csv)

Using io.StringIO is neat if you do not want to write an actual file but send the CSV as API response or via email for example.


Finding a decent headset is harder than expected

posted on Monday 18th of May 2020 in ,

I am a big fan of Sennheiser. Their headsets in various price segments perform extremely well. They might not be the best of the best in every segment, but they are constantly great. Their customer service is one of the best I have ever dealt with and it shows that the company is and has been focused on what they do for decades – and yet I am wearing a Bose NCH700 on a daily basis.
Let me tell you a story of broken bluetooth stacks, hipster marketing and awesome customer support.

I spend a lot of time on calls and video conferences. While more recently I actually got more time to write code, five to ten hours a week are still spent talking to people. (Which is actually a big improvement over the previous 30 or so.) I actually prefer to wear a headset during calls, for some reason it is easier for me to understand multiple people on a call with headphones. Also having a slightly better microphone than what is built into the device I use to dial in helped a lot with the Cherry MX blue switches in the best keyboard ever. (There, I said it. Fight me IBM Model M fans 😉 )

Usually I use my iPad for calls. Video conferencing software on macOS hovers somewhere between ‘broken’ and ‘spyware or malware like behavior’. iOS and iPadOS do a slightly better job at keeping them in check. I also do not have to hear my fans spin up because Hangouts decided 100% usage on a core is required for a voice only call. But from time to time I share my screen with my IDE and terminal so instead of joining from two systems I only use my Mac. This gives us a short, but decent list of requirements for the headset:

I did not add „amazing sound“ to the list. This is not my primary use case, but it is a soft requirement, considering that I also travel on a regular basis and would prefer to not hate every single second of watching a movie with them. At the same time I do not expect them to compete with headphones on an amp created to sound amazing.

The gold standard for connectivity on Apple devices are clearly AirPods. Switching devices takes one click. ONE! It will not get better than that. I liked the original AirPods and I really like the AirPod Pros. Not for hours on conference calls, but for listening to podcasts or taking a call walking the dog or listening to music while running.

The first headset I bought was the Sennheiser PXC 550 Wireless. It is a comfortable headset, sound is good, the touch controls are as bad as you expect and it folds nicely. I noticed something strange though – from time to time the headset announced „call ended“ while I was very much still on a call. I had to turn the headset off and on to reconnect. Which means I usually missed 20-30 seconds if the device I used did not have speakers to fall back to. Some more testing revealed that this happened during longer calls, after roughly 30 to 40 minutes.

Well, that has to be broken, right? It took me three minutes on the phone with Sennheiser to arrange getting them replaced. (I would not be able to get past the stupid voice assistant that replaced IVRs in that time when calling other companies’ support hotlines.)

Sadly the problem persisted. Having the same issue with my MacBook Pro, iPad Pro and iPhone lead Sennheiser CSR and me to the conclusion that the headset somehow does not want to play nicely with Apple products. So they upgraded me to the Momentum Wireless for free. (Did I ever mention that I love Sennheisers customer support?)

The Momentum has a unique design which you either love or hate. But it is as comfortable and the sound without direct comparison might be a little bit better than the PXCs, but likely the is the same. It does not fold as nicely as the PXC, but still good enough to have it in a carry on. But again: Same issue.

Calling Sennheiser again they then dispatched me to one of their engineers. Some troubleshooting later they told me they will call back in a bit. A day later I was told they can reproduce the problem and think Apple broke the Bluetooth stack in their operating systems. I am not ruling this out as an option per se, but since I have not heard half of the Internet complain about it, it somehow seems unlikely. Maybe someone else messed Bluetooth up? Using a USB Bluetooth adapter was not really an option for me, so their only known work around would not do the trick. At that point I was actually looking for a new gaming headset, so we agreed to exchange the Momentum for an GSP 670 and call it a day. The GSP is okay, but might be less comfortable depending on your head shape and if you wear glasses. Also you look stupid wearing it. Just do not put a mirror above your gaming monitor and you should be fine.

Now that my goto vendor was not an option anymore I was a bit lost. I know about the other brands you would usually compare Sennheiser to, but do I really want to go down that route? The Sony WH-1000XM3 and Bose QuietComfort 35 II are two headphones you likely know or at least have heard of. They are everywhere and in every comparison. And people seem to be really happy with them. After talking to a friend who spends a lot of time reviewing hardware two unexpected options were added to my maybe list: Microsoft Surface Headphones and Beats Studio 3.

At this point my thought was „Instead of messing with Bluetooth I just get the Beats with Apples W1 chip. Allegedly now well sounding headphones that just work“. Oh boy. Oh boy, was I wrong. Most headsets I listed so far are very similar. One might have a specific software feature you might love. One might have a sound profile you prefer. But overall you can buy any of them and be happy. Except the Beats. They belong in a trash can.

The Studio 3 does not fold well. The ear cups made my ears sweat after 20 minutes of use (I wasn’t even aware that that is possible). Enough pressure on my head to make them slightly uncomfortable. And for some reason switching between devices like you would expect with a W1 chip did not work properly. I really tried to not be picky about the sound, but it was as bad as their reputation suggests. Zero out of ten, would not recommend. Back to Amazon.

Time was in my favor though: The Bose NCH700 were just released and a local shop nearby had them available. At first it felt like they put a bit much pressure on the top of my head. After wearing them in it was gone. They are comfortable. I like the sound. The software works. The microphone is doing a good job not picking up the barking dog and noise cancellation works well enough that I do not even know she is barking. They do not fold as nicely as the Sennheiser, but at this point I’m willing to live with that.

I have been using the Bose for roughly seven months now and they just work. Not a single problem since I got them. They are paired with my iPad and Mac Pro, announce both connections and automatically switch to the device playing back something or joining a call. The biggest challenge for me was leaving Sennheiser behind and accepting that Bose are indeed more than something you buy for your airline miles because you do not need anything else from the catalogue. If anything would happen to them I would buy them again without thinking about it twice.


Fenix for Twitter

posted on Sunday 26th of April 2020 in , ,

I have a love hate relationship with Twitter. I like the concept, but I have a certain disliking of how the platform evolved. One thing that makes Twitter usable for me – and likely one of the reasons why I am still actively using it – is the option to use a third party client. The best feature of all third party clients? Tweets ordered by time, not by some broken algorithm. More recently Fenix seems to gain a lot traction and press, so I took a few days to check it out.

In case you never used a third party client a few things upfront: Twitter put some technical limitations in place and you will not be able to use all features, like polls and moments. Practically I never really cared. I have an experience without every second tweet being sponsored and I can properly mute keywords and people.

So far I have been a Tweetbot user and really happy. Push for messages and DM works, but you do not get any notification if people retweet or favorite a tweet of yours. Can you imagine a world without instant notification that you gained arbitrary Internet points? It is quiet. I like it. I use a few lists to separate people I want to follow but sometimes fall into a rage-like state throwing 90 tweets about a broken graphic cards fan in my timeline – sorry, but I won‘t read this in the morning when I try to catch up with what‘s going on. I also have a separate list for esports to avoid spoilers on game days – really handy, especially since Overwatch League casters (and players) are the most spoiler happy people I have ever seen.

After putting Fenix on my iPhone and iPad and logging in I had to force quit the app for it to actually load my timeline. Well, things like this can happen, whatever. I started out on the iPhone and a few things immediately felt right.

The design is clean and modern. The default font and color choices are nice but Fenix gives you some option to customize it to your liking. The app just feels optimized by someone using Twitter a lot or has a good understanding of user interface design. I do not want to give Tweetbot a too hard time when it comes to the UI, but there are simply a few things that feel outdated or not well thought out.

The button to send a tweet for example is at the bottom right for example instead on the top right minimizing the travel of your finger quite a bit and making it possible for people with smaller hands to send a tweet one handed.

Swiping right you can reply to a tweet or swiping a bit further you can quote the tweet. Swiping left you can favorite, retweet or bookmark. Yes, you can bookmark tweets. I love this feature. I often want to keep one or two tweets to follow up on later and I either have to dig through my favs or open them in Safari and add them to my read it later list.

There are a few significant differences which you will notice quickly. There are no push notifications. There is no background refresh. You open the app and you pull to refresh. That‘s it. You start reading tweets. The same goes for notifications. You go to the tab to check if there are any. If you like to pretend missing features are a stone in the road to minimalist nirvana you will love this app.

Lists are basically implement the worst possible way on both platforms. You read that right, the developer actually ships different UI concepts based on the screen size. Something I wish more apps would do.

Both on the iPhone and iPad you have multiple columns you can swipe through horizontally. This is a super nice concept for the iPad to keep your eyes on multiple things – the last time I used a client that supported this feature that nicely was before Tweetdeck was bought by Twitter. The iPhone displays one column at a time. The problem is the only way to access a list is making it a column on the iPad. If you have many lists this will not be great interaction model. On the iPhone you also have a modal window. To consume content. This is just bad and somehow feels even worse than columns. It is like the client is telling you: „Here is the list. Do not get too comfortable here, it is not what you want to focus on“.

Talking about the iPad app I have to vent a bit about the status bar. It is not a solid color and the separator line for the side menu and the left most column goes all the way up – through the date and time. Somehow this is more distracting that I ever imagined.

There is also an Android version of the app, but sadly no macOS one. While I do not often use Twitter on my Mac I like the fact that Tweetbot exists and keeps my timeline in sync. The iPad app feels well enough designed to use Catalyst and bring it to the desktop. The multi column concept should work pretty well on larger screens.

Talking about sync: totally broken. Neither the read status of my timeline nor my bookmarks are synchronized. This is actually the dealbreaker for me. I often switch between my phone and my tablet and I surely will not try to manually figure out where I stopped reading.

Fenix is still relatively young compared to other clients like Tweetbot or Twitterrific, but it shows a lot of potential and is actively being developed. If you are only using one device the broken sync will likely not bother you much and if the third party client limitations of Twitter are something you can live with I think Fenix should be the first one to check out when you are looking for a solid client improving the Twitter user experience tremendously. Secretly I hope this will give our friends at Tapbots a bit more heat to step up their game UI / UX game. It has been long enough, it is time for a paid upgrade to Tweetbot 6.