Securing the enterprise by securing APIs (Google Cloud Next ’17)
Articles Blog

Securing the enterprise by securing APIs (Google Cloud Next ’17)

afternoon, everybody. My name is Chris Latimer. Thanks for joining us here. I know that we’re the only
thing standing between you and drinks at the
cocktail reception, so we definitely appreciate
your attendance for our session here, today, on securing the
enterprise with secure APIs. I just now realized that rhymed. I never actually said
it out loud before. My name is Chris Latimer. I’m one of the field
engineers from Apigee, now part of Google, and
I’ll let my cohorts here introduce themselves
later on, when they get to their part of
the presentation as well. Really excited to be
here and talk to you guys today about this topic. Being Apigee, we
absolutely love APIs. And so what do we
mean when we say APIs, when we talk about
APIs and API security today? Are we just talking about
the technical aspects of API, what protocols we’re using,
system to system integration– that type of thing? The answer is no. We really think about
APIs more broadly. We think about them from
a technical perspective, but we also think about them
from a business perspective. And later on, we’ll get to
hear from Ticketmaster and some of the interesting
things that they’re doing from an API
perspective and how they’re securing their APIs. So if you look at what all
we’re doing with APIs today, it’s hard to go even
more than five minutes without using an
API of some form. If you check social media,
you’re using an API. When you checked into your
flight, checked the flight status, that’s another API call. When you make a
purchase online, that’s also going to be
another API call, so of course, we need
to secure these APIs. Very, very important
data and functionality are running through
APIs, so we often want to make sure that
we’re going to secure those, and our first
reaction is to think about something like this. We’re thinking about, well,
that’s a lot of valuable data that somebody could
get access to, that maybe they
shouldn’t get access to, and so what we’re
going to do is we’re going to lock this down
as tightly as possible. Put everything in the vault.
Don’t let anybody get access to this stuff. Put in place all the
monitoring, all the tools that we can to really lock
this important capability down. And the problem with that
is that, if all of your API security starts
to look like this, you’re missing out on a lot of
the value that APIs can bring. So the hard part of
API security isn’t just locking everything down. If we just wanted to
have a secure API, we could just not
build any APIs. That’s the most
secure API of all. But putting together a
secure API while still being able to foster innovation
inside of your organization, avoid putting in place
unnecessary delays, red tape, bureaucracy– doing all that while
keeping your API secure is a much, much harder
problem to solve. And so how do we do that? And in order to think
about this problem, I like to put it into the
context of imagine that you are just creating a new account
online– you see a new site; you want to register
for an account– and the way that we would
think about this process. If the site wants your
email, sure, why not? I give this out all
the time to any site. Its pretty usual
that they want that. They want my address. Not a big deal to give
that information out. It’s usually pretty
easy to find. You start to get to something
like a phone number. Well, now, I’m a
little skeptical. Am I going to get robocalls? Are you going to text me? Is this for two-factor auth? Is there a good reason for it? And if they say something
like, oh, what’s your social? Well, now, you’re going
to slam the brakes on. Almost nothing is going to get
access to that information. You might think about
it a little bit. Is there a legitimate
need for me to give this information out? There might be in certain cases. You’ll look and see, is
this actually the site that it claims to be, or
is this a broken SSL cert? And if you’re comfortable
with the risk reward, you’re comfortable with
handing out that information. An API security requires
a similar approach. We’re going to expose
different types of APIs, and they’re going to have
different types of security requirements that we’re going
to want to impose on those APIs. So let’s look through
a few examples here. Let’s imagine that you’re
working for a health care company, maybe a
hospital, and you want to create an
API that’s going to expose hospital locations
and hours, public information. There’s nothing
sensitive about this, so we can just
open it up and just let everybody go use
this thing like crazy. Well, in a sense, yes,
but there are still things we need to be concerned about. For example, this may be part of
a mission critical application for us. This may be what– you know when you
call up the hospital, and you ask where the
nearest location is, somebody might be using the same API. When you go to the website,
you may be using the same API. So if all of a sudden,
you open this up and somebody starts sending
a flood of traffic your way, that can still cause pretty
major problems for you. So we need to think about that. The traditional approach
to this is let’s put in place
heavyweight governance. Let’s lock this down. Let’s not give anybody
access unless we go through a rigorous process
to understand how you’re going to be using this
API, how many calls are you going to send to this. But a better approach to
that is traffic management, so we can put in place things
like quotas and spike arrest and so forth to
limit our exposure while still maintaining
that culture of fostering innovation. Another thing that
we’ll usually do is try to put into place
a service registry. Let me figure out exactly
who’s using which API and make sure that we’re
tracking this information very, very closely. A better approach of that is to
use something like an API key, where we can do
this automatically. So rather than have a
registry that’s probably going to get out of date and out
of sync and give us unreliable information, what we can
do is we can use an API key to automatically make sure
that each application that’s making an API call is
authorized to do so. We also have to worry about
content based attacks. Even though this API may not
expose sensitive information on purpose, if
somebody starts sending in SQL injections or
cross-site scripting or those type of
things, we have to still be cognizant of the
payload and make sure we’re validating that payload. And a common way of
approaching this problem is with an API gateway, and
because I came from Apigee, I’m going to happen
to mention, we just have a really good
API gateway that solves a lot of these
problems that you have that we’re talking about today. Now, I promise this is not going
to be a commercial for Apigee, but I did just want
to mention this. So if you want to
talk more about Apigee and its capabilities,
we are happy to do so. So let’s think
about another API. Let’s think about one
that, all of a sudden, is opening up scheduling
or doctor availability. So now, we’ve got
a few more things that we have to think about. We have to start thinking about
we don’t want anybody just registering for
access to this API and starting to create
appointments for doctors. We want to have a
process in place that lets us vet the partners who
are going to be using this API, and the traditional
approach to this is to have a manual process,
the same type of thing. We’re going to have a form that
somebody’s going to fill out. We’re going to take
this information. We’re going to decide whether or
not you get access to this API. A better approach
is self-service, so rather than you answer a
bunch of questions about how your API works, instead of
you looking through and having conversations and meetings
and slowing things down and coordinating schedules,
we can have a developer portal that’s going to give you the
ability to give your partner self-service aspects– self-service capabilities to
register for access to an API, see the documentation, and
start to use it right away. We are also going to need
things authentications, so we’re going to need identity. We’re going to need to make
sure that the person that is registering for a– or that is scheduling
an appointment is the person that’s the
patient in our system. So we need some way
of identifying folks, and the traditional approach
is going to be direct user authentication. If I have partners who are
going to be authenticating my patients, I don’t necessarily
want them to have access to that patient’s credentials. Instead, a better approach is to
use delegated identity and user authorization, so
we would have things in here like OpenID Connect,
social logins, other logins where our partners can redirect
that user back to our identity management solutions so
that they can sign in, and then we know that
it is actually them without the partner
ever actually seeing those credentials. Now, what if we start to get
into some really tricky areas? What if we’re going to
expose patient data? We’re going to give information
about their medical history. What do we have to
think about then? Well, now, we have
to think about making sure we’re not end up
in jail because we’re breaking a bunch of laws. So there is a time
and a place that we need to have very,
very locked down, tight security in our APIs. The point of this
presentation isn’t to say that we absolutely don’t
need to think about tight API security. We do, and there are times when
that’s absolutely appropriate. So we want to make sure that
the person who’s looking at this data is authorized. Is it your primary
care physician that should have access
to this information, or is it somebody from
marketing purposes that wants to sell you a drug? They probably shouldn’t have
access to that information. We want to be able to
have an audit trail. We want to be able to
look back and see who has looked at this over time. At one of the hospitals
that I’ve worked with before have had issues where
celebrities have come in, and people have wanted to kind
of see what’s going on there. Obviously not
something that you want to have happen in a HIPAA
compliant environment. And then there are all
the basic things, too. Of course, we need to
still think about TLS. We need to use HTTPS. We’re going to want to
look at payload encryption in certain cases to make sure
that our data is very, very tightly locked down. And so from a
general perspective, these are some of the things
that we need to think about from an API security– as we’re designing
our API security. And with that, I’m going to
hand it over to my colleague Greg Brail, who is going to talk
about a few specific use cases and case studies that
we’ve seen in the wild. GREG BRAIL: Excellent. Thanks Chris. So I’m going to talk
now a little bit– Chris has talked
about the context. Why do we want to select
different types of API security? When is it important to
have everything locked down to the max versus making
things flexible developers? But what I really want
to also talk about is get into a little more detail
about API security and ways to secure APIs. And the reason why I think it’s
important to talk about this– if you’re a web
developer or have even come close to web
development, and you work for any sort of
moderately large company, you probably had to go through
some sort of mandatory training where they taught you about all
the things that can go wrong with web app development. And you learned about CSS
attacks and SQL injection and all kinds of
things like that. We don’t have that for
APIs, and APIs are a little different than web apps. There are some things that
can happen to web apps that don’t happen to APIs. There are some
things that happen to APIs that you don’t
think about as much when you’re building a web app. So I wanted to make sure we
talk a little bit about API security, and one of
the ways I like to do it is by talking a little bit
about some of the things that can go badly when
you don’t think about all the aspects of API
security that are important. So for instance, if you remember
nothing else, everything that has a URL– or if I’m really
fancy, I’ll say URI– has an API. So Chris said earlier, one
way to make an API secure is not to have one. That’s true, but if you built a
mobile app, and the mobile app talks to some service on the
internet, you have an API. You might not have
advertised it, developers. You might not have a
Swagger spec for it. You might not have GitHub
repo with sample code. But you have an API, and
a reasonably smart person can, at home, jiggy
up their Wi-Fi router to see what’s going on
between your app and your API and figure out what’s happening. And a bunch of smart
people have done this to embarrassing effect. For instance, the highly
critical enterprise API run by Kylie Jenner
was broken to by a 19-year-old
who looked at it and discovered that his Kylie
Jenner mobile app, or actually her website, had
no API security, and he could get some
interesting information. Similarly, here’s
a great article I found on Programmable
Web that came out about a year ago titled “How to
Use Undocumented APIs to Hack Your Tesla.” That’s something that, if you
own a Tesla, it’s kind of cool. Hey, I can hack my Tesla. But the truth is that these
are all cases of when people actually were able to look
in and see what was happening between the client and the
server and figure out the API. So when we ignore the
fundamentals of API security because we don’t
have an API, you have to really ask yourself,
well, do we have an API? Maybe if everything is inside
one room or one LAN, OK, we don’t have an API. But otherwise, we do, and
as enterprise networks get larger, as companies
get larger, as things like– as companies have
global networks, everything is effectively
an open network nowadays, so you have to assume that
when you design your apps. So why are APIs important, and
why is API security important? Well, a really great example
I can talk about here is the world of banking. Especially in the
US, a lot of people use services like
Mint or Yodlee, which will do aggregation of
their bank account information. I use Quicken a lot. It has a similar feature. And if you use any
of those services, the first thing they do was
say great, give me your bank account username and password. I should have used one of
Chris’s disturbed person images for that one because, once
someone says give me your bank account username
and password, it means what they’re
going to do is they’re going to store
your bank account password. Yeah, they’re going
to encrypt it, but they’ve got to decrypt
it in order to use it, and then they’re going to screen
scrape your bank’s website. They’re going to
make HTTP requests as if they were a web browser. They’re going to parse
the HTML, and they’re going to figure out what
your bank is telling them. So basically, some
of the banks have started to realize that this
is not great for security because when their
end users are giving their passwords to
these third parties, you don’t know what the third
parties are doing with them. You don’t know if the
third parties are storing that data in a safe way. Even if they’re storing
that data in the safest way possible, they still have
access to their end users’ unencrypted passwords, and
those passwords give you access to do everything
you can do from your bank’s website, including transferring
large sums of money, making big stock trades,
whatever you can do. If we had took an API
approach, on the other hand, where banks were
to offer APIs that used a security mechanism such
as OAuth, that would mean that, as an end user, you would
not give these third parties your password. You would instead go through
an authentication process with the bank that gives
the third party token, and that token can now
be set up by the bank to only offer access to
these specific things that the bank wants
them to have access to. So for instance, JP
Morgan Chase realized this and wrote something in
their annual report, and they got into a
little bit of trouble with the “New York
Times” that said [MOCKING] Jamie Dimon wants to
protect you from innovation. [NORMAL TONE] And
he has a point, and they have a point, too. But the idea here is by offering
open APIs with well-known API security techniques
such as OAuth, we can have a much, much
better handle on security because security is not
about eliminating all risk. Security is about managing risk. And by giving
third parties a way to have access to
important information like financial information
in a controlled way, where you’re only allowed to do
what the bank allows you to do, we’re in a much better
position than we are today, where you give
someone your password, and they can do anything. So I’m now going to
talk about five or six specific API security things
and kind of use some examples. So one big API security
mistake we can make is to not use TLS, or SSL
as it used to be called, and now the protocol
we’re supposed to use today is actually called TLS. So for instance, a
security researcher did a study last year,
and he discovered that, around the world,
especially in South Asia, at the time, something like
30% used non encrypted links. Two years ago, that was 90%. Now, maybe that’s
not your password, but we’re to the point
now on the internet where everything that is even
slightly important, and in fact, even everything that isn’t,
should be encrypted with TLS. That’s why, when
you do a Google search, is encrypted with TLS. The cost is just not there. But it’s more than just,
hey remember to use TLS, don’t put an unencrypted
API on the internet. It’s how you use TLS. So for instance, in some
frameworks for the client, it’s very easy to turn
off TLS verification because you’re
doing development, and it’s kind of a pain to
get the right certificate, and the name on the DNS record
doesn’t match the certificate, and we’re in a hurry. So we’ll just check that
box in Node.JS or PHP or Java or whatever
that says don’t verify the TLS certificate. That opens it up
for your client app to be able to have its
supposedly secure web traffic be intercepted by
anyone in Starbucks who has control over the
Wi-Fi or over the DNS router. TLS verification is
tremendously important. Not only that, TLS has
a bazillion options. What version are you using– 1.0, 1.1, 1.2? What cipher suite are you using? Are you using Elliptic Curve? What encryption
algorithm are you using? And these things
change all the time. So as anybody who’s offering
an API or any web app and is using TLS,
you have to remember to keep up with what’s
happening in the security world, follow the best practices
from your security people and your company and
elsewhere, and make sure that you’re continually
evaluating how you’re using this technology and
keeping up with the latest attacks. For instance, I still
see developers sometimes occasionally want to
write code, and they want to hash something,
and they use MD5. MD5 has been a no-no in the
security world for years. In fact, at Google, it’s
not even in the code base. They took it out. But people still remember
this from a few years ago. So this is why it’s very
important, anytime encryption is used, to really
think about this. So now, we move on to some
of the other basics of APIs. Chris talked about APIs
that require an end user to be authenticated. There are all kinds of things
that can go wrong with this. So for instance, the Nissan
LEAF a little while ago came with a mobile
app, and the mobile app let you do things
to your car like turn on and off
the climate control or start your car remotely. And to make it easy for
people to use the app, the authentication
mechanism for the app required you to put
in the car’s VIN number, which is a unique
ID that’s on every car. It’s actually on the
dashboard of your car, so it’s not a secret, and it
is structured in such a way that every type of car, most
of the digits are the same, and only the last
five digits vary. So you can imagine that
if you knew that, hey, I can turn on the
climate control on one Nissan LEAF using a VIN
number, what if I were to cycle through 10
or 15,000 VIN numbers and see if I can do that other
Nissan LEAFs in the world? And a security researcher
named Troy Hunt figured out that the answer was yes. Now, I’m not saying this
to make Nissan look bad, because they fixed this. I’m saying this to
give you an example of what happens
when you don’t think through API authentication. There are a lot
of best practices out there for basic things,
like signing users in, checking passwords, resetting passwords. It’s actually pretty
hard to get right. Companies like Google
do a good job of it. Smaller companies that are
specialized in the security world do a good job of it. Large companies that
don’t, don’t always do a great job of it. I’ve seen some
horrible password reset practices from various
websites I use in my life. These kinds of
things can show up as massive API security
vulnerabilities if you’re not careful. But it’s not just about
authenticating the end user. In the world of
APIs, one thing we do that we did not do
in the world of web apps is we also authenticate
the application. If you were to ask me what
does API management do– because I am an
API management guy. That’s what we do at Apigee. API management is about
managing the relationship between the developers who
build applications and the APIs that they use, and
a big part of that is managing those
security credentials. OAuth, which is the de facto
standard for API security now– and in fact, from a
lot of security now– has built into it the concept
of application credentials. In order to build an application
that gets an OAuth token– say they use a
Google Cloud API– you not only have to supply
some user credentials, but you’ve got a supply
application credentials, which the application developer gets
as part of the development process. Now, the application
of credentials are not 100% proof
against attacks. I mean, anybody who has
access to the application can use those application
credentials, But what they do is they give the security team a
lot of extra layers of defense, and that’s what security
is really about. So for instance, we have had
cases of Apigee Edge customers who had– we had one who accidentally
pushed a version of an app to the App Store
that had a bug in it. And it was actually hitting
an authentication API in an infinite loop,
hundreds of times a second, and it was soon overwhelming
their whole technology stack. Now, if they had a different
set of application credentials for every version and
platform of that app, they actually would have
been able to cut that app off instantly in their
API management tier, or redirect traffic
from that app to some sort of other server,
which is able to mediate it. And as a matter of fact,
they didn’t quite did that, but they did have the
application authenticated, so they were able to
take some of those steps. But that’s an
example of why it’s important to not only
authenticate the end user but the application. This also gives
you an opportunity to do some things
like analytics, and we’ll talk a little bit
later about bot detection. But knowing which application
is making an API call is one of the pieces
of information you can use in that world. Now, I mentioned OAuth. OAuth has been around
for a couple of years. It’s become the de facto
standard for API security. And basically, one
reason why it works is because what it
allows you to do is it allows a client who
makes an API call to exchange some credentials for a token,
and that token gives you access to the API. And with OAuth, it’s possible
for the API provider, using software like Apigee or
many other bits of software, to control what a
token is allowed to do. Furthermore, unlike a
password, a token uniquely identifies a single
application on a single device. It’s not just a generic bit
of security information, and OAuth tokens
can be refreshed. But OAuth is a fairly
complex family of specs. They are IETF standards. There are a lot of smart people,
including some of our security folks at Google,
working on it, but there are lots of ways to use OAuth. So for instance,
one way to use OAuth is to make an API call where
you send in your password and get back the token. Another way to do it is to have
the user’s device open up a web browser window where they can
get a token by being redirected to a website, and that website
can do additional things like check cookies, ask
for a second factor, do all kinds of
things like that. Now, it turns out that the
OAuth folks have actually been working on a spec
that describes, hey, these are the best practices for how
to use OAuth in native app. We recommend using this web
browser redirect method. Here’s how to do it safely on
different mobile platforms. Furthermore, there
have been a couple of things that have been found
as vulnerabilities in the OAuth specs that have been fixed in
some of the newer IETF specs. So it’s really important, if
you’re going to use OAuth, to pay attention to
some of the requirements that you see in the community
and some of the discussion that goes on, especially in
the IETF, because there are a lot of smart
people who have written a lot of
really good stuff about how to use
OAuth correctly. I should also mention
authorization, and in this form
of authorization, I mean once you’ve
authenticated a user, and you know who they are,
what are they allowed to use. It’s easy to make
some dumb mistakes. This one security
researcher discovered that his bank actually– I don’t think it
was a very big bank, but they actually were not
verifying the account numbers on the back end. So he figured out the API
that his mobile app was using, and he was able to transfer
money from any account to any other
account in the bank. Now, that’s a huge mistake. That’s like people
getting fired mistake, but these kinds of
things can happen. Developers don’t always
think of these things. One of the things
I like about APIs is that an API is a
contract, and it’s exactly clear what
is supposed to be sent between the
client and the server. So unlike a web app, where
you’re downloading JavaScript and executing who knows
what, a provider of an API can specify exactly what’s
sent, exactly what’s received, what the valid responses are. That means you can write test
plans and regression tests and penetration
tests, and you can verify that what you
think your API is, is what you actually
implemented. I think that’s actually
an advantage of APIs, and it’s an advantage
of API security. And by having that kind of
spec, and by thoroughly testing that spec, you should
be able to prevent these kinds of dumb
authorization mistakes that– they’re easy to laugh
at now, but there but for the grace
of God go all of us. Right? Another one, this is really
important, is rate limiting. So the best example of API
rate limiting I could find came from a researcher that
did some work a couple of years ago on Snapchat when they were
a much smaller and less mature company. He discovered that,
given a phone number, it was actually possible to
get information about someone’s Snapchat profile. Now, that doesn’t sound
like a big problem, but it turned out that there
was no rate limit on this API, and he was able to send in
huge batches of phone numbers. Now, if you could imagine say
Beverly Hills, California– I guess people use
cell phones now, but even then, it doesn’t have
that many phone numbers in it. It wouldn’t take
very long to get a lot of interesting
information that way. So this was, to me,
the best example of, in the real world,
an API that should have had some sort of rate
limit to help with security. And I mention this because
pretty much any API that could be subject
to a brute force attack is something that will be
subject to a brute force attack. And it may be subject
to a brute force attack by someone who is just
hammering it for fun, but it could be somebody
with legitimate credentials. Lots of people can sign up
for these services, so things like product IDs,
user names– if you can put in someone’s phone
number and get information. Another great example is
that OAuth password grant type, where you send in your
password in an API call, and you get back to
a security token. Well, if there’s no
rate limit on that, now you have a distributed
password cracking API that people can now use to
crack your password database. So it’s great to monitor,
and this stuff is important. But by having basic
rate limits on APIs– like no one should call this API
more than 500 times a second, or a given
application is limited to a certain number of APIs
calls per day or hour or month, or even possibly introducing
artificial delays to some APIs that may be
called very frequently. Those make a very,
very big difference in the kind of security
posture you have and prevent these kinds of attacks. And also we’re going to finally
mention probably the biggest security list of
things that developers need to know about,
comes from OWASP. It’s called the Top 10. I couldn’t find
anything since 2013, but basically, they
list the top threats to applications on the internet. Interestingly enough, they
don’t all apply to native apps that use APIs. If you’re building APIs,
some of the JavaScript things and cross-site scripting things
don’t really apply to you, but the rest of it does. Injection attacks,
misconfiguration of security, bad authorization decisions–
a lot of the things that I’ve talked
about are in here, but there are some things I
didn’t talk about as well. For instance, the
number one attack is an injection attack, so
classic SQL injection stuff, like can I put
semi-colon delete star from customers in my user name. It applies to APIs just as
much as it applies to web apps. Yes, there are things you can
do with regular expressions and stuff to protect from
this, but it’s really, really important that, as
application developers, we remain just as cognizant
of this in the world of APIs as we do in the
world of web apps. So to sum up this
part, and then I’m going to talk for two
or three more minutes and turn it over to Ismail. I think everyone needs to know
some basic things about API security, and this is not
necessarily the same things you need to know for
web app security. And a big part of
it, in my opinion, is understanding the
importance of things like a rate limiting, traffic
management, content attacks, and basically, the
idea that anything that can be called
with a URI, someone is going to figure out
what your app is doing and try to break it. And as app developers, we have
to be very cognizant of that. At the same time,
by designing APIs that follow the right
security principles, by making smart choices about how much
security we need on an API. And by making sure that our API
definitions are clear and well understood, we’ve now
designed something that we can test that has a much
smaller surface area than a web app. And I would argue, if
you follow all the rules, and you do everything right,
its possible to make things with APIs that have fewer
security risks than what we see today with web apps. Now, I’m going to talk for just
a short time about Apigee Edge. This is the product that has
been acquired by Google as part of Google Cloud,
and basically, what we do in Apigee is we
provide a product that lets us insert ourselves
into the world of APIs. So many companies, when they’re
looking to offer an API, they’re asking themselves
how do I implement a lot of these things? How do I implement OAuth
and connect it to my users– to my existing user
authentication system, my existing single
signon system? How do I implement a rate limit? How do I do checks, basic
threat protection checks like malformed JSON and XML. These are the kinds of things
that we’ve always been built into the Apigee Edge product. And the idea of Edge is
that, as an API provider, you can put the API
management product– and there are a lot of
products in this category– in front of your API, and it can
handle a lot of these aspects, so that your back end,
the stuff your developers are building– you don’t have to
build this into every service. The larger the company
you are, the more it makes sense to
have one place where you apply many of
these security policies as opposed to building them
into every single application. Many large companies–
Google has an API management for its own APIs. All the big companies
that have APIs have API management as
part of their software stack at the edge
of the network, somewhere between the load
balancers and firewalls and the stuff that
actually runs the APIs. What Apigee Edge is–
it’s a product that allows you to configure it
to be that API management layer for the API and configure
it however you need it to work. Now, I’m going to
talk for two minutes here about Apigee
Sense, and this is the last thing I’m
going to talk about, which is a thing that we’ve
used to extend Apigee. And this is going to segue
nicely to what Ismail’s going to talk about, is we’re going
to talk a little bit about bots. So there are all kinds of bad
things that can happen to APIs. People who are not
legitimate users may try to hammer
your API, to crack it, to brute force it
just to mess with you. But what can also happen, which
is a little bit more subtle and harder to deal with, is
when legitimate users who have got legitimate application
and end user credentials use your APIs in an
illegitimate way. This happens a
ton, for instance, in the travel industry,
where, gee, every hotel has the same prices. Well, it’s because
somebody somewhere is screen scraping or botting
the APIs of all the other hotel websites and trying to book
a room in every single price category to try and find
out what the prices are and what the availability is. There’s a lot of bots
that happen in retail. Ismail comes from Ticketmaster. Some of you may have
used Ticketmaster. They have some
experience with bots, as much as they’re
able to tell us. So Apigee Sense is a product
that uses the fact that we collect a lot of data in the
Apigee platform about the APIs that we handle, and basically,
we can use that information– it’s information about
IP addresses and things– to figure out and to
score incoming API calls on whether or not we
think they might be a bot. Some of this is heuristic,
like for instance, American retail companies
don’t have a lot of customers in Eastern Europe. Or some of it is
heuristic like iPads tend not to be located
in Amazon data centers. Some of it is machine
learning, and some of it is standard big data stuff. Was one OAuth token used
to make more APIs calls than a human user could
have made using an app? Was one OAuth tokin used
to scan through APIs, like to scan through all the
phone numbers in Beverly Hills, from 0000 to 99999? These are the sorts of
things that you can see, and by looking for
patterns like this, we’re able to then
score incoming API calls as definitely a bot, like
we know this IP address is a bot because everybody is
getting bot traffic from it, to probably not a bot,
to something in between, and then take action. Sometimes, take action
is block the API call, but that lets the bot people
know that you’re blocking them. So sometimes a better action
is to let the back end know so it can provide a different
result, a different price, a different availability, even
slow down the API traffic. So that’s kind of what
we’re looking at doing. So with that, that’s
really all that I have to say about Apigee. I’m now going to
turn it on to Ismail, who is an SVP at
Ticketmaster and is going to talk to us about his
experiences with APIs and API security. Thank you very much. ISMAIL ELSHAREEF:
Thank you, guys. Oh, whoa, it’s loud. Hi, everyone. Thank you, Apigee and
Google, for having me here. This is really
exciting to be here to talk to you guys about
our experience with APIs from a business perspective. I’m going to talk a little
bit about the tech involved as well. So at Ticketmaster, we’re all
about delighting global fans in a distributed world. A little bit about myself
before I jump into that. I’ve been in this battlefield
of tech and product for a while now. I started as a software
engineer, back end engineer and front end engineer. I’ve been doing APIs
for the last seven years or eight years. Joined Ticketmaster 18 months
ago to open up the platform and open up their APIs,
and six months ago, I started running the entire
product organization– consumer product organization,
so very, very exciting. So show of hands– who’s been on Ticketmaster
the last six months? Tried to buy tickets? I can’t see. Not a lot, actually,
which is surprising. So keep your hands up. Keep it up if you were
not able to buy tickets when you visited Ticketmaster. I don’t need this. I don’t need to talk. I don’t need to talk. It’s all good. It’s happening. Oh, we have one person. OK, I’m here for you, man. I’m here for you. So we have problems. Not every fan that comes
to could buy tickets. So that’s why, when
we started, what really got me excited about
the opportunity at Ticketmaster is the leadership there– Jody Mulkey, the CTO,
and the entire team– is that we’re really
reinventing Ticketmaster. We’re changing the way
we’re selling tickets, and that’s by having
a fan first thinking. From our infrastructure,
and completely changing and upgrading our
infrastructure– we have an incredible
partnership with AWS and Amazon where we’re moving our
private cloud into AWS, and it’s a process that
we’re working on right now– into opening our
APIs, and that’s a partnership that we have with
Apigee that’s been fantastic. So infrastructure opening up
the APIs, and then our products had to be built with
fans in mind first, which is not traditionally the way
it’s been done in the past. And what that means is that
really looking at all our fans, different personas,
and understanding why we’re building the
products that we’re building. And our promise to our fans is
that we are unlocking moments of joy for global fans
through unforgettable product experiences. So if you think about that, and
thinking about the fan and fan first, you’ll see that– and I don’t know if you
guys follow Ticketmaster– we have been changing the
way we offer value to fans. So for example,
this happened today. It’s a pre-sale for Ed Sheeran,
who’s coming to North America. It’s going to be one of the
biggest tours this year. His new album dropped a week
ago, and it’s killing it. But this way is for us to
say, hey fans, preregister, and we will text you a
link before the presale where you can buy
Ed Sheeran tickets. We did it with 1975, John Mayer,
Depeche Mode– you name it. Because what we
find is that, when fans come to Ticketmaster for
Depeche Mode, for example, who– they haven’t toured
in seven years, and if you can get
tickets, you’re going to freak out because
I will freak out, too, because I’ve been wanting to go. And I signed up, and I got– yes, we have a fan in the house. I love it. But it’s real. No one’s going to drop $4,000 to
go see Beyonce at the Rose Bowl if they’re thinking logically. So it’s an emotional
business that we’re in. We just want to make sure that
fans are not disappointed, are not turned down, and
are actually getting the tickets that they want. We see 72 million
unique visitors a month. Not all of them are fans. So we have a real,
real need here, and a real problem to solve
with some of the challenges we run into. So tickets are
the only commodity I can think of that
go up in price– up in value once you buy them. I don’t know if you guys bought
tickets to Hamilton recently. I have not seen it. I can’t afford it. But we deal with a lot of
issues and a lot of problems. Bots are one of them. Limit violations–
people are trying to circumvent the limits
we put on an event. So as a fan you cannot buy– for certain events, you cannot
buy more than eight tickets. For some events, you cannot
buy more than four tickets, just to make it fair and give
access to a lot of other fans that want to go see the act. And some people and some
bots and some bad actors trying to circumvent that by
they want to get, I don’t know, 100 tickets. I’m talking about concerts
now, and theater, not sporting events per se. So there’s that level of
complication and challenge that we have, and then
the biggest challenge we have is reserve and do not buy. So during an on
sale, when people bad actors come and select
a pair of tickets or a bunch of tickets, and they
don’t actually end up buying them, what happens
is that when they actually select them, those tickets
are put into a hold, and no other fans can get
to them, which is bad. We have ways of going
around that, obviously. We’ve been doing
this for a while now, so we have our own algorithms
and our own systems that can kind of determine or try to
glean whether this traffic is coming from a good actor, a real
fan, a bad fan, a robot, a non robot, and all of
that good stuff. But that’s just, Ticketmaster apps. That’s just our closed system. But in order for us to
deliver on our promise to our fans, our global
fans, we determined that we have to open up the platform. We have to make our APIs
available to other developers, to other partners,
clients, where they can build
applications or can build experiences
for their own fans, so everyone gets access
to those tickets. So when I started 18
months ago, what we did was we consolidated all of
these different platforms that we have into one
simplified, very easy to use platform partnered with
Apigee to expose those APIs. And they would start
using those APIs across the board in all
different channels starting with our own apps and
their own properties, but also for pre-sales
on fan clubs, for example, last minute
channels, you name it. So we have different
outlets for those APIs. But the reason we
wanted to go this route, obviously to open up
the platform, but also for any measurement of security
that we apply to one channel now, we apply it to the
APIs, and that’s it. We don’t have to think
about it anymore. Because when you
apply it to the APIs, it doesn’t matter
who’s using the API. The same level of security
is going to be applied. In our world, what Greg
said, that security is not eliminating risk. It’s mitigating risk,
and for us, it’s all about mitigating risk. It’s all about mitigating risk. We’ll never know, in real
time, who’s a true fan and who’s a bot. So if you’re like me, if you
go to a show once a week– and I pay out of
pocket, by the way. I don’t get free tickets. If you know how to get
free tickets, let me know. My behavior and my
pattern of activity can look like a
bot to our system because I’m coming from
my home wifi trying to buy tickets to John Mayer,
to 1975, this and that. And it’s like, oh, my god,
this guy or that entity is buying a lot of tickets,
so maybe it’s suspect. We’re trying to make
it smarter and better, but true fans and super fans
can come across as bots, and that’s a problem that
we’re trying to solve. But we opened up the platform. We have a portal, We created an ecosystem where
you can go to the portal and understand what APIs
we have, who’s using them, and how you can
get access to them. We really followed all the
best practices and everything that Apigee advocates
and puts out there. And from my previous experience
as well, all the best practices that I learned
the hard way sometimes, that we should be
doing in order for us to have a successful
API program. So we’ve done all of this stuff. We integrated with some
incredible partners to get access and to reach other
fans that might not necessarily come to And as a result of that
opening up the platform and really thinking
broadly, we’ve seen, just last year, that 10%
of all tickets sold by us came off of platform,
came through partners, came through our APIs,
outside of our products. Fascinating, right? But guess what– the
challenges still remain. The challenges that we’ve
seen in our own products still remained true in
the open platform world. They just got amplified. Actually, almost
all the errors we see in the Apigee portal
coming through our APIs are of these two
kinds of errors. Too many requests, rate
limiting– so people trying to make too
many API requests, trying to get access to
those tickets, reserve them. And because we
have rate limiting, it’s preventing our system from
going down and overwhelming our system, but also
really controlling the API access for that
particular user or application. But also look at the
unauthorized calls. A lot of people are
trying to make calls without being authorized,
without the right OAuth token. It’s a real problem. And because building an API– a lot of people think, oh,
it’s a simple interface, and I have the SDK
and documentation. It’s pretty simple. Right? It actually takes a lot
of effort and planning, especially when it comes to
security, identity management, who has access to what, error
messaging, all of that– it takes a village for the
API to actually be useful and to be simple. That’s the challenge
we run into, so when we focus on the fan
and opening up everything and trying to think how do we
make that access of this API that we just put out
there favorable to fans, and for developers to build
applications that actually let fans through. So the way we did it was
really follow the basic best practices. I cannot emphasize that. A lot of times, the
biggest problems happen because you don’t follow
the guidelines and the best practices, like authorization,
authentication, OAuth, rate limiting, using bot
blocking that Apigee provide if you don’t do this
for a living like we do, SSL, just the basic stuff. And if you do the
basic stuff, you’re actually 50% there, like Greg
and Chris were talking about. In our case, we really had to
think about the reserve API call. That’s where actually people
put real tickets, real value, on hold, so we need to rethink
how that API call actually works. So the way we did it was
we’re introducing friction. We’re trying to
introduce friction. It’s like you cannot
do that anonymously. You authenticated
your application. We have the OAuth token. Great. But now, we want to know
who’s the actual user making that call, because
then I’ll have extra information about
that user that will help me determine is that
really a legitimate fan, or is that a bot
that’s trying to– that hacked the app, and
now using it in a bad way. So introducing friction
by forcing registration or signing in, using whatever
you want to sign in for. We use Ticketmaster
Accounts for that. And using the fan score,
which is our algorithm to determine whether you’re
a good actor or a bad actor– basically, a fan or a bot. And the last thing,
obviously, which is really, really important– and we just did
that with Apigee– is identity management,
entitlements. So when somebody’s making an API
call with their OAuth token– so if that checks out,
that means Apigee says, OK, this OAuth token is good. Now, taking their API
key and their identity and pinging our system, saying
what role does this player have in our system? Are they a supervisor? Are they a fan with a bunch of
tickets that they can transfer, and all of that stuff? And once you get that back,
you add a bunch of headers to the call, and then when
we get it in the back end, we understand if this action is
permitted by this user or not. So if you want to
transfer some tickets– now just because you’re a
fan doesn’t mean anything. You have to be a
fan and also have the rights to
transfer those tickets or to sell those tickets
or to cancel an order. So these are the three
learnings that we found. Just opening up an API ecosystem
and a platform actually works really well. Is it perfect? It’s not. It’s never going to be
perfect, and we’re always learning every single day. But the fact that we
sold 10% of our tickets off of platform using our
APIs that we opened up just 18 months ago is a huge
testament to the power of APIs to business, but also
the fact that you have to stay vigilant in order
for you to really expand. I mean, we haven’t scratched
the surface with APIs just yet. And with that, I thank you for
being here, and we’re hiring. We want people to help us
with this transformation that we’re on. And I guess I’ll
open up to questions. [MUSIC PLAYING]

One thought on “Securing the enterprise by securing APIs (Google Cloud Next ’17)

Leave a Reply

Your email address will not be published. Required fields are marked *

Back To Top