A little while back I wrote a few blogs on the possibilities of background threads in Legato. For this week we are going to discuss using a Legato background thread to monitor the SEC’s RSS feed of filings. The script checks the SEC’s RSS feed and if a specific CIK is found in the feed, a message box showing the filing information is opened. This script could be adapted to work with any RSS feed, but as it stands the script can help you see when your company’s filings are disseminated to the public.
Friday, October 12. 2018
LDC #106: RSS Monitor
If you haven’t read my previous blogs on background threads, I would suggest doing so. Much of the code for this script is about thread management, and since that has been covered previously, I will not discuss these parts in detail. If the script looks eerily similar to the Running in the Background script, that is because I used it as a basis for this one. The first major modification was changing the thread control to use session variables (as discussed in Running in the Background Part 2).
Here is the complete script. Like the previous background thread scripts, this script file will need to be saved before it is run.
// // // GoFiler Legato Script - RSS Monitor // ----------------------------------- // // Rev 10/19/2018 // // // (c) 2018 Novaworks, LLC -- All rights reserved. // // Monitors RSS activity // /********************************************************/ /* Global Defines */ /********************************************************/ int start (int f_id, string mode); int stop (int f_id, string mode); int main (); int setup (); int background (); #define GROUPNAME "RSSMonitor" #define SVAR_STOP "Stop" #define SETTING_CIK "CIK" #define SETTING_WAIT "Wait" #define SETTING_CHECK "Last Checked" #define RSSFEED "https://www.sec.gov/cgi-bin/browse-edgar?action=getcurrent&CIK=&type=&company=&dateb=&owner=include&start=0&count=40&output=atom" /********************************************************/ /* Hook Support */ /********************************************************/ /* Functions in this section deal with the hooks to */ /* control the background thread. */ /********************************************************/ /************************************************/ /* Hook Globals */ /************************************************/ /****************************************/ handle hBGScript; /* Background Script Handle */ /****************************************/ int setup() { /* Called from Application Startup */ /****************************************/ string fnScript; /* Us */ string item1[10]; /* Menu Item */ string item2[10]; /* Menu Item */ int rc; /* Return Code */ /* */ /* ** Add Menu Item */ /* * Define Functions */ /* o Start */ item1["Code"] = "EXTENSION_START_RSS_MONITOR"; /* Function Code */ item1["MenuText"] = "&Start RSS Monitor"; /* Menu Text */ item1["Description"] = "<B>Start RSS Monitor</B>\r\r " + /* Description */ "Starts monitoring RSS feed for specific items.";/* of function */ /* o Stop */ item2["Code"] = "EXTENSION_STOP_RSS_MONITOR"; /* Function Code */ item2["MenuText"] = "Sto&p RSS Monitor"; /* Menu Text */ item2["Description"] = "<B>Stop RSS Monitor</B>\r\r " + /* Description */ "Stops monitoring RSS feed for specific items."; /* of function */ /* * Setup */ SetSessionValue(GROUPNAME, SVAR_STOP, FALSE); /* Set Stop Variable (initialize it) */ fnScript = GetScriptFilename(); /* Get the script filename */ /* * Check for/add menu item */ rc = MenuFindFunctionID(item1["Code"]); /* Look for existing */ if (IsError(rc)) { /* Not already added */ rc = MenuAddFunction(item1); /* Add the item */ if (IsNotError(rc)) { /* Was already be added */ MenuSetHook(item1["Code"], fnScript, "start"); /* Set the Test Hook */ } /* end error */ else { /* Failed? */ return rc; /* Leave with error */ } /* end failed */ } /* end not added */ /* * Check for/add menu item */ rc = MenuFindFunctionID(item2["Code"]); /* Look for existing */ if (IsError(rc)) { /* Not already added */ rc = MenuAddFunction(item2); /* Add the item */ if (IsNotError(rc)) { /* Was already be added */ MenuSetHook(item2["Code"], fnScript, "stop"); /* Set the Test Hook */ } /* end error */ else { /* Failed? */ return rc; /* Leave with error */ } /* end failed */ } /* end not added */ return ERROR_NONE; /* Return value (does not matter) */ } /* end setup */ /****************************************/ int main() { /* Initialize from Hook Processor */ /****************************************/ /* */ /* ** Initialize Hook Processor */ /* * Check State */ if (GetScriptParent() != "LegatoIDE") { /* Not running in IDE? */ return ERROR_NONE; /* Done */ } /* end not in IDE */ /* * Set up */ setup(); /* Add to the menu */ return ERROR_NONE; /* return */ } /* end function */ /****************************************/ int start(int f_id, string mode) { /* Start Hook Processor */ /****************************************/ /* */ /* ** Run Start */ /* * Safety */ if (mode != "preprocess") { /* Not preprocess? */ return ERROR_NONE; /* Return w/ no error */ } /* end not preprocess */ if (hBGScript != NULL_HANDLE) { /* Already Running? */ return ERROR_NONE; /* Return w/ no error */ } /* end already running */ /* * Cleanup last run (if any) */ SetSessionValue(GROUPNAME, SVAR_STOP, FALSE); /* Set Stop Variable */ /* * Start it */ hBGScript = RunBackgroundScript(GetScriptFilename(), /* Run Background Script */ "background"); /* name of the function */ if (IsError(hBGScript)) { /* Failed? */ MessageBox("Couldn't start thread 0x%08X", GetLastError()); /* Tell user */ } /* end failed */ return ERROR_NONE; /* return */ } /* end function */ /****************************************/ int stop(int f_id, string mode) { /* Stop Hook Processor */ /****************************************/ int ix, /* Index */ rc; /* Return Code */ /* */ /* ** Run Stop */ /* * Safety */ if (mode != "preprocess") { /* Not preprocess? */ return ERROR_NONE; /* Return w/ no error */ } /* end not preprocess */ if (hBGScript == NULL_HANDLE) { /* Already Running? */ return ERROR_NONE; /* Return w/ no error */ } /* end already running */ /* * Stop it */ SetSessionValue(GROUPNAME, SVAR_STOP, TRUE); /* Set Stop Variable */ for (ix = 0; ix < 10; ix++) { /* Loop */ rc = WaitForObject(hBGScript, 100); /* Wait for it */ if (IsNotError(rc)) { /* Worked? */ hBGScript = NULL_HANDLE; /* Clear Handle */ MessageBox('i', "Stopped Thread."); /* Tell User */ break; /* Stop */ } /* end worked */ } /* end loop */ if (ix == 10) { /* Failed to Stop? */ MessageBox('x', "Failed to stop thread."); /* Tell User */ } /* end failed to stop */ return ERROR_NONE; /* return */ } /* end function */ /********************************************************/ /* Background Thread */ /********************************************************/ /* Functions below this point run in a seperate script */ /* engine. Therefore global variables will exist but */ /* will not have the same values as above. */ /********************************************************/ /****************************************/ int background() { /* Background Function */ /****************************************/ string props[]; /* Properties */ string tmp, /* Temp String */ cik, /* CIK to find */ results, /* Results */ ldate; /* Last Check Date */ handle hFeed; /* RSS Feed */ int cnt, /* Count */ ix, /* Enumeration Index */ wait, /* Wait Time */ rc; /* Return Code */ /* */ /* ** Run Background Function */ /* * Load Settings */ /* o Wait Time */ tmp = GetSetting(GROUPNAME, SETTING_WAIT); /* Get Wait Time */ if (tmp == "") { /* Empty or Error? */ tmp = "60"; /* Set to 60 Seconds */ } /* end empty or error */ wait = TextToInteger(tmp); /* Convert to integer */ if (wait < 10) { /* Wait is small or error? */ wait = 10; /* Set to at least 10 seconds */ } /* end wait is too small */ wait = wait * 10; /* Want to wait in smaller intervals */ /* o CIK */ cik = GetSetting(GROUPNAME, SETTING_CIK); /* Get CIK */ /* o Last Check */ ldate = GetSetting(GROUPNAME, SETTING_CHECK); /* Get Last Check Date */ if (ldate == "") { /* No Date? */ ldate = "1980-01-01"; /* Set to something */ } /* end no date */ /* * Main Loop */ StatusBarMessage("Started RSS monitor."); /* Tell User */ cnt = -1; /* Clear Value */ while (TRUE) { /* Loop forever */ /* o Sleep until time to check */ if (cnt >= 0) { /* Not first time through */ for (ix = 0; ix < wait; ix++) { /* Loop for wait time */ Sleep(100); /* Sleep for 100ms */ if (GetSessionInteger(GROUPNAME, SVAR_STOP) == TRUE) { /* Want to Stop? */ break; /* Stop */ } /* end want to stop */ } /* end loop for wait time */ } /* end not first time through */ if (GetSessionInteger(GROUPNAME, SVAR_STOP) == TRUE) { /* Want to Stop? */ break; /* Stop */ } /* end want to stop */ /* o Load Feed */ StatusBarMessage("Loading RSS..."); /* Tell User */ hFeed = RSSLoadFeed(RSSFEED); /* Load Feed */ if (IsError(hFeed)) { /* Failed to Load? */ cnt = 0; /* Reset Count */ continue; /* Next */ } /* end failed to load */ if (GetSessionInteger(GROUPNAME, SVAR_STOP) == TRUE) { /* Want to Stop? */ CloseHandle(hFeed); /* Close Feed */ break; /* Stop */ } /* end want to stop */ /* o Read Feed */ results = ""; /* Clear Result */ cnt = RSSGetItemCount(hFeed); /* Get Count of items */ for (ix = 0; ix < cnt; ix++) { /* For each item */ StatusBarMessage("Checking RSS (%d of %d)", ix + 1, cnt); /* Tell User */ if (GetSessionInteger(GROUPNAME, SVAR_STOP) == TRUE) { /* Want to Stop? */ break; /* Stop */ } /* end want to stop */ props = RSSGetItemProperties(hFeed, ix); /* Get Properties */ if (StringToDate(props["PublicationDate"]) < /* Date is before */ StringToDate(ldate)) { /* last check? */ break; /* Leave loop */ } /* end date is older */ if (cik != "") { /* Have a CIK? */ if (FindInString(props["Title"], cik) < 0) { /* Not Found? */ continue; /* Not relevant? Skip */ } /* end not found */ } /* end have a CIK */ if ((GetStringLength(results) + /* Add Entry is */ GetStringLength(props["Title"])) > 900) { /* going to be too long */ results += "\n..."; /* Add to Results */ break; /* Stop */ } /* end log */ if (results != "") { /* Already added something */ results += "\n"; /* Add New Line */ } /* end already added something */ results += props["Title"]; /* Add to Results */ } /* end for each item */ if (GetSessionInteger(GROUPNAME, SVAR_STOP) == TRUE) { /* Want to Stop? */ CloseHandle(hFeed); /* Close Feed */ break; /* Stop */ } /* end want to stop */ if (results != "") { /* Had Something? */ MessageBox('i', results); /* Tell User */ } /* end had something */ /* o Save Check time */ props = RSSGetItemProperties(hFeed, 0); /* Get First item */ if (ldate != props["PublicationDate"]) { /* Different Date? */ ldate = props["PublicationDate"]; /* Set Last Check Date */ PutSetting(GROUPNAME, SETTING_CHECK, ldate); /* Save Setting */ } /* end different date */ /* o Reset Feed */ CloseHandle(hFeed); /* Close Feed */ StatusBarMessage("Ready"); /* Reset Bar */ } /* end loop */ /* * Clean up */ StatusBarMessage("Stopped RSS monitor."); /* Tell User */ return ERROR_NONE; /* return */ } /* end function */
I will not go over the setup and main functions in detail as they are similar to many of the blog posts we have done in the past. The main function runs setup, so you can hook the script using the Run ribbon command in GoFiler instead of having the script saved to your extensions folder. The setup function adds the required menu hooks. The only special line worth mentioning in the setup function is where we call the SetSessionValue SDK function to initialize our session variable to a value for the first time. This ensures that later any other calls to get the session variable will not fail.
Let’s look at the start and stop functions. These functions are called by the menu hooks to start and stop our background thread. In many ways they are similar to our past blog posts, but for thread control the functions now use session variables.
/****************************************/ int start(int f_id, string mode) { /* Start Hook Processor */ /****************************************/ /* */ /* ** Run Start */ /* * Safety */ if (mode != "preprocess") { /* Not preprocess? */ return ERROR_NONE; /* Return w/ no error */ } /* end not preprocess */ if (hBGScript != NULL_HANDLE) { /* Already Running? */ return ERROR_NONE; /* Return w/ no error */ } /* end already running */ /* * Cleanup last run (if any) */ SetSessionValue(GROUPNAME, SVAR_STOP, FALSE); /* Set Stop Variable */ /* * Start it */ hBGScript = RunBackgroundScript(GetScriptFilename(), /* Run Background Script */ "background"); /* name of the function */ if (IsError(hBGScript)) { /* Failed? */ MessageBox("Couldn't start thread 0x%08X", GetLastError()); /* Tell user */ } /* end failed */ return ERROR_NONE; /* return */ } /* end function */
We check to see if the background thread is already running. If it is, we can just leave the function. If the thread isn’t running, we set our session variable. We do this using the SetSessionValue function. We have defined a name for the variable so it is easily changed if needed. This sets the “Stop” variable to false. Now we can start our background thread using the RunBackgroundScript function. Remember that this function needs a script filename for a parameter, so your script must not be untitled. Let's look at stop next.
/****************************************/ int stop(int f_id, string mode) { /* Stop Hook Processor */ /****************************************/ int ix, /* Index */ rc; /* Return Code */ /* */ /* ** Run Stop */ /* * Safety */ if (mode != "preprocess") { /* Not preprocess? */ return ERROR_NONE; /* Return w/ no error */ } /* end not preprocess */ if (hBGScript == NULL_HANDLE) { /* Already Running? */ return ERROR_NONE; /* Return w/ no error */ } /* end already running */ /* * Stop it */ SetSessionValue(GROUPNAME, SVAR_STOP, TRUE); /* Set Stop Variable */ for (ix = 0; ix < 10; ix++) { /* Loop */ rc = WaitForObject(hBGScript, 100); /* Wait for it */ if (IsNotError(rc)) { /* Worked? */ hBGScript = NULL_HANDLE; /* Clear Handle */ MessageBox('i', "Stopped Thread."); /* Tell User */ break; /* Stop */ } /* end worked */ } /* end loop */ if (ix == 10) { /* Failed to Stop? */ MessageBox('x', "Failed to stop thread."); /* Tell User */ } /* end failed to stop */ return ERROR_NONE; /* return */ } /* end function */
The stop function is very similar. If the thread is already stopped, we can leave. Otherwise we set the “Stop” value to true and then wait for the thread to end using the WaitForObject function. If it takes more than 10 seconds to stop, we leave with a MessageBox letting the user know that the thread may still be running. Chances are if this happens, the thread is likely waiting for a response from the SEC’s website and will end after the website responds to the request.
As you can see, these functions are very similar to past blog posts. Instead of using a file to control the thread, though, we are using session variables. Session variables are a lot more reliable and make the code much easier to understand.
With these functions out of the way, we can focus on the meat of the script: the background function.
This function uses the StatusBarMessage function, which normally displays progress to the user. Since most computers are extremely fast, it is unlikely these messages will be seen, but they can be useful for debugging purposes.
/****************************************/ int background() { /* Background Function */ /****************************************/ string props[]; /* Properties */ string tmp, /* Temp String */ cik, /* CIK to find */ results, /* Results */ ldate; /* Last Check Date */ handle hFeed; /* RSS Feed */ int cnt, /* Count */ ix, /* Enumeration Index */ wait, /* Wait Time */ rc; /* Return Code */ /* */ /* ** Run Background Function */ /* * Load Settings */ /* o Wait Time */ tmp = GetSetting(GROUPNAME, SETTING_WAIT); /* Get Wait Time */ if (tmp == "") { /* Empty or Error? */ tmp = "60"; /* Set to 60 Seconds */ } /* end empty or error */ wait = TextToInteger(tmp); /* Convert to integer */ if (wait < 10) { /* Wait is small or error? */ wait = 10; /* Set to at least 10 seconds */ } /* end wait is too small */ wait = wait * 10; /* Want to wait in smaller intervals */ /* o CIK */ cik = GetSetting(GROUPNAME, SETTING_CIK); /* Get CIK */ /* o Last Check */ ldate = GetSetting(GROUPNAME, SETTING_CHECK); /* Get Last Check Date */ if (ldate == "") { /* No Date? */ ldate = "1980-01-01"; /* Set to something */ } /* end no date */
We start by loading the settings for the RSS monitor. This includes the amount of time we want to wait between checks, the CIK to watch for (if any), and the last time the RSS monitor read the feed, which is important because if you stop the monitor and restart it, you don’t want to be notified of filings you have previously seen.
We get the wait time and if it is empty or below 10 seconds, we adjust the value. I decided to use 60 seconds as the default wait time as it is a nice, round number. Keep in mind that if the wait time is too long, the script isn’t as useful, and if the wait time is too short, the SEC might think you’re trying to break their feed. This is also why the wait time has a minimum amount of 10 seconds. We multiply the wait time we’ve chosen by ten. I’ll discuss why we do this later. We get the CIK and the date of the last check. If the last date does not have a value, we just set it to a value that is unlikely to occur in the feed.
Now that we have all our settings loaded, we can start the main loop of the thread. Remember, since we want our thread to run until we tell it to stop, the background function must have a loop.
/* * Main Loop */ StatusBarMessage("Started RSS monitor."); /* Tell User */ cnt = -1; /* Clear Value */ while (TRUE) { /* Loop forever */ /* o Sleep until time to check */ if (GetSessionInteger(GROUPNAME, SVAR_STOP) == TRUE) { /* Want to Stop? */ break; /* Stop */ } /* end want to stop */ /* o Load Feed */ /* o Read Feed */ /* o Save Check time */ /* o Reset Feed */ } /* end loop */ /* * Clean up */ StatusBarMessage("Stopped RSS monitor."); /* Tell User */ return ERROR_NONE; /* return */ } /* end function */
For our loop we are using a while(TRUE) loop. We could also done while the “Stop” variable is false but since as we add code to the loop we will be checking “Stop” more often, it does not matter much. The main sections of the loop have been set up in the comments but for now contain no code. Instead our loop waits until the “Stop” variable is true. Once this happens, the loop breaks and the thread ends. We can run the thread in this state, but it is not a very good idea. The thread contains no functions that block processing (such as I/O functions) or calls to the Sleep function, so the loop will use all available processing time. Let’s start by adding our wait.
... /* o Sleep until time to check */ if (cnt >= 0) { /* Not first time through */ for (ix = 0; ix < wait; ix++) { /* Loop for wait time */ Sleep(100); /* Sleep for 100ms */ if (GetSessionInteger(GROUPNAME, SVAR_STOP) == TRUE) { /* Want to Stop? */ break; /* Stop */ } /* end want to stop */ } /* end loop for wait time */ } /* end not first time through */ if (GetSessionInteger(GROUPNAME, SVAR_STOP) == TRUE) { /* Want to Stop? */ break; /* Stop */ } /* end want to stop */ ...
Since we don’t want to wait the first time the thread enters the loop, we will reset the cnt variable to -1 outside the loop. Then in the loop if cnt is greater than or equal to 0, we will wait. We could wait for the entire wait time using the Sleep function. The problem with this approach is when we tell the thread to stop, it may take up to wait in order for the thread to respond. Asking the user to wait for 60 seconds? Ouch! So instead we will wait in a number of smaller increments and check the “Stop” variable each time. Our wait time is in seconds so we could Sleep in one second intervals. But even that may not be very responsive to the user. So we multiplied the wait value by ten and instead will wait in 100ms intervals. This makes the thread wait for the wait time but it will stop almost immediately if asked.
Now that our wait is out of the way, we can start working with the RSS feed.
... /* o Load Feed */ StatusBarMessage("Loading RSS..."); /* Tell User */ hFeed = RSSLoadFeed(RSSFEED); /* Load Feed */ if (IsError(hFeed)) { /* Failed to Load? */ cnt = 0; /* Reset Count */ continue; /* Next */ } /* end failed to load */ if (GetSessionInteger(GROUPNAME, SVAR_STOP) == TRUE) { /* Want to Stop? */ CloseHandle(hFeed); /* Close Feed */ break; /* Stop */ } /* end want to stop */ ...
We start by using the RSSLoadFeed function. This function takes the address to an RSS feed XML file in RSS or ATOM format. It returns a handle to an RSS Feed object, which we can use to get information about the feed. If the feed doesn’t load for some reason, we can reset count (so that our wait time triggers) and then just continue the loop. Since this function may take some time to complete, we check the “Stop” variable before continuing in case the user wants to stop. Let’s move onto reading the feed.
... /* o Read Feed */ results = ""; /* Clear Result */ cnt = RSSGetItemCount(hFeed); /* Get Count of items */ for (ix = 0; ix < cnt; ix++) { /* For each item */ StatusBarMessage("Checking RSS (%d of %d)", ix + 1, cnt); /* Tell User */ if (GetSessionInteger(GROUPNAME, SVAR_STOP) == TRUE) { /* Want to Stop? */ break; /* Stop */ } /* end want to stop */ props = RSSGetItemProperties(hFeed, ix); /* Get Properties */ if (StringToDate(props["PublicationDate"]) < /* Date is before */ StringToDate(ldate)) { /* last check? */ break; /* Leave loop */ } /* end date is older */ if (cik != "") { /* Have a CIK? */ if (FindInString(props["Title"], cik) < 0) { /* Not Found? */ continue; /* Not relevant? Skip */ } /* end not found */ } /* end have a CIK */ if ((GetStringLength(results) + /* Add Entry is */ GetStringLength(props["Title"])) > 900) { /* going to be too long */ results += "\n..."; /* Add to Results */ break; /* Stop */ } /* end log */ if (results != "") { /* Already added something */ results += "\n"; /* Add New Line */ } /* end already added something */ results += props["Title"]; /* Add to Results */ } /* end for each item */ if (GetSessionInteger(GROUPNAME, SVAR_STOP) == TRUE) { /* Want to Stop? */ CloseHandle(hFeed); /* Close Feed */ break; /* Stop */ } /* end want to stop */ if (results != "") { /* Had Something? */ MessageBox('i', results); /* Tell User */ } /* end had something */ ...
First we will use the RSSGetItemCount function to get a count of items in the RSS feed and reset our results string. Now that we have the count we iterate over the feed and check the entries. We check the “Stop” variable in this loop as well to ensure our thread is responsive. Then we use the RSSGetItemProperties function to get information about the current item. This function returns an array of named strings that have the properties of the item. The ones we are interested in are the “Title” and the “PublicationDate”. Using the StringToDate function, we check the “PublicationDate” against our ldate variable, which represents the date of the last item we read. If this item is older than the last time we read something, we can stop processing the list of items.
Now that we know that the current item is new, we can check to see if it is relevant by checking for the cik in the “Title”. If we don’t have a cik we assume the item is relevant. If we still want this item we can add it to a results string. There is some extra code here to make sure the results string doesn’t get too large because we are going to display the information in using the MessageBox function which has a limit on the size of the contents.
Once we are done reviewing the items we do one final check to see if the thread needs to stop, and if not we display the results, if any, using the MessageBox function. Once we’ve displayed the results we can move on to preparing for the next check.
... /* o Save Check time */ props = RSSGetItemProperties(hFeed, 0); /* Get First item */ if (ldate != props["PublicationDate"]) { /* Different Date? */ ldate = props["PublicationDate"]; /* Set Last Check Date */ PutSetting(GROUPNAME, SETTING_CHECK, ldate); /* Save Setting */ } /* end different date */ /* o Reset Feed */ CloseHandle(hFeed); /* Close Feed */ StatusBarMessage("Ready"); /* Reset Bar */ ...
We use the RSSGetItemProperties function to get the properties of the first item again. In RSS, generally the first item is the latest item. We will use this to our advantage and take the “PublicationDate” date of this item as the last item we read. We will also use the PutSetting function to write this date to an INI file. This means next time the feed will pick up here. Finally, we reset the feed for next time by closing the handle with the CloseHandle function.
With that our script is complete. If you want to change the CIK for which the script searches, you can run the thread once and it will create an INI file in the your user’s application data folder named the same as the script file. In that INI file you can add a line that is CIK=000000000, where 0000000000 is the CIK you wish to watch. This handy script allows you to see when your company’s filings are disseminated on the SEC’s website. Like most examples on this blog, with a little modification you could have this script monitor any RSS feed for any information you want, which opens the door to many possibilities.
David Theis has been developing software for Windows operating systems for over fifteen years. He has a Bachelor of Sciences in Computer Science from the Rochester Institute of Technology and co-founded Novaworks in 2006. He is the Vice President of Development and is one of the primary developers of GoFiler, a financial reporting software package designed to create and file EDGAR XML, HTML, and XBRL documents to the U.S. Securities and Exchange Commission. |
Additional Resources
Legato Script Developers LinkedIn Group
Primer: An Introduction to Legato