I am currently finishing up efforts on my largest integration yet and I wanted to write about the design, moving parts and lessons learned. In this first part of the series, I will discuss the high level plan and overview of what components in ServiceNow I plan to use. I’ve never seen this approach used exactly before, but it seemed to be the best way to tackle my issue and hope for feedback.


The Goal

  • Move away from consuming .csv file on FTP for our user/location data
  • Centralize sensitive data feeds of user profile data with REST.
    • How many times do multiple systems want to get a user feed from HR? We will aim for a REST endpoint to gather the data and make endpoint available for many apps (this is outside of ServiceNow scope). Build smarter.
  • Avoid cloud platform to cloud platform data integration where possible.
    • It is challenging to centrally control your data when cloud apps are sending data wild west style to each other outside of your network

2017-07-23 14_21_39-ESB myID Integration - Visio Standard


The Approach

The first challenge was to think about which approach to take with ServiceNow’s REST toolset.

Option 1) REST direct to table

You can write directly to the user table using REST. This is great in theory but has its disadvantages.

2017-07-23 14_45_15-ESB SN Documentation v0.1 - WordMinimal Customization – All the heavy lifting is done by the side sending the data
2017-07-23 14_45_31-ESB SN Documentation v0.1 - Word Minimal Flexibility – Challenge to manipulate the data if needed, what they send is what you get
2017-07-23 14_45_31-ESB SN Documentation v0.1 - Word Reference Fields – This approach expect the sending system to populate Reference Fields with sysIDs. Unless its reasonable to provide the sender with a hardcoded set of sysIDs with very little variation, this is always a problem.
2017-07-23 14_45_31-ESB SN Documentation v0.1 - Word Expected Field Names – Your sending system must provide the fields with the exact expected ServiceNow field names

Example

POST https://[instance].service-now.com/api/now/table/sys_user

 

Option 2) Import Set Rows & Transform Maps

My normal go-to, Import Set Rows & Transform Maps give you excellent functionality and tool set built for integrations

2017-07-23 14_45_31-ESB SN Documentation v0.1 - WordModerate Customization – You will have to receive data, stage it and map it to desired fields. Flexibility provided with field level scripting and transform map scripting.
2017-07-23 14_45_15-ESB SN Documentation v0.1 - WordModerate Flexibility – You can manipulate date before writing to the final desired table(s). Most use cases requiring scripting can be satisfied here.
2017-07-23 14_45_15-ESB SN Documentation v0.1 - Word Reference Fields – This approach expect allows you to receive non-sysIDs and easily populate reference fields
2017-07-23 14_45_15-ESB SN Documentation v0.1 - Word Expected Field Names – You can choose the name of the fields that the sending REST message will be including, allowing you to tailor to their needs.
2017-07-23 14_45_15-ESB SN Documentation v0.1 - WordBasic Error Handling – A lot of stuff is already built for you as far as error handling, such as responding with rejection if expected data is not present.
2017-07-23 14_45_31-ESB SN Documentation v0.1 - WordMost useful with “flat” data – If your sender is providing data sets within multiple levels/nodes of JSON, it takes a bit of scripting to step through each node for each field.
2017-07-23 14_45_31-ESB SN Documentation v0.1 - WordVersion Control – Unlike more mature REST endpoints, you cannot contain multiple versions of your integration endpoint should it evolve overtime and especially if you have multiple integrations on the same endpoint where some may upgrade before others.

For the most part, Option 2 has always done a good enough job for me. It’s error handling was convenient though it left some to be desired. The version control is a great feature but many integrations fair fine when there is only 1 other system involved.

 

Option 3) Scripted REST

The final option is a more advanced framework with maximum flexibility, though you may find yourself having to build out all aspects.

2017-07-23 14_45_15-ESB SN Documentation v0.1 - WordPotentially most of the benefits of Option 2, however…
2017-07-23 14_45_31-ESB SN Documentation v0.1 - Word.. Everything is just a little more manual work to build. Build your own Response Status Codes and Error Handling from scratch. Write the code to do all field name grabbing, data manipulating, reference field look ups and more.
2017-07-23 14_45_15-ESB SN Documentation v0.1 - WordFull control of Response Codes – Your success 200 code, internal error 500, etc. Here you can dictate in your code exactly what status codes to return.
2017-07-23 14_45_15-ESB SN Documentation v0.1 - WordVersion Control – Allows v1 and such so that multiple integrations using the same endpoint can coexist as you develop.


 

The Decision

Are you still reading this? Great, then you must be on the edge of your seat wondering… which did I choose?

1c652ee4-cfcc-46de-8f7f-dd5cbb0f11aa_560_420

I chose… Option 2 and 3!

Yea – I know, quite the season finale twist if I do say so.. which, I did.

Why was that? Well, for my client I had an extra complexity. Because I have 2 potential data sources sending me data to the same end point at the same time, I have to sort through the data as its received to determine “Do I accept only data set A, data set B, or a combination of each?” While this could be done in Option 2 with lots of script.. it leaves me in a position where I’m doing all most as much work for 2 as I would 3, but not getting the advantages of option 3.

And so I decided to use Option 3 to do my first level checking, while feeding that into Option 2 to give me nice troubleshoot mechanics that are built into ServiceNow plus versioning and custom error handling.


 

The Design

The goal is to take two endpoints sending me user data from slightly different sources. Most fields have same, but each source has fields and data unique to only its set. I need to accept it, figure out which data set I’m dealing with and put those fields into my import set row. Then, based on the results, I will build a response status code as appropriate. I am also putting the heavy lifting of the Scripted REST code into the Script Include. Finally, we are going to do this all in a scoped app because that’s just the kind of thing we like to do to ourselves to learn.

Integration Detail 01

 


The Wrap

We’ll call it a break here for Part I. When we return, we will examine a message as it flows through this design and point out interesting pieces that I picked up while building and testing it.

Thanks for following me through this overview – any comments are much appreciated and I look forward to continuing this.

Advertisements