Last week, Dave added to our Bulk Filing script, giving it the ability to add folders and to detect if it’s going over the maximum allowable size from the SEC. This week, we’re going to do our final update on this script and develop its ability to actually file to the SEC. As part of this, we’re going to read back the SEC’s response, determine if it’s an error or a success, and if it’s a success we’ll parse the accession notices out and create an output file that maps the accession number to the original file. This will be very handy in case one of the filings comes back suspended, so you can just look up the accession number of the file to see which one failed.
Friday, January 18. 2019
LDC #119: Bulk Filings Part 4
This is a pretty straightforward change, where we really only need to modify the code of the run function. Let’s look at the differences below:
// Start of File
hAttachment = PoolCreate();
WriteLine(hResult, OPEN);
test = ReplaceInString(LIVE_FLAG, TEST_DIRECTIVE, live);
bulkType = "BulkTest";
if (live == "LIVE"){
bulkType = "BulkLive";
}
WriteLine(hResult, test);
// Write Attachments
First, at the very beginning of the file before we do anything else, we need to set the new bulkType variable’s value. This will be used in submitting the file to the SEC. If the filing is a LIVE file or a TEST file, this value will be different.
// Finish File
WriteLine(hResult, ATTACHMENTS_CLOSE);
WriteLine(hResult, CLOSE);
CloseHandle(hResult);
ProgressClose();
if (limit >= 0) {
LogSetMessageType(LOG_WARNING);
AddMessage(log, " Bulk filing size (%d) exceeds EDGAR transmission (%d) limit.", GetFileSize(hResult), SEC_FILE_LIMIT);
AddMessage(log, " Remove files after entry %d (%s).", limit + 1, files[limit]);
LogSetMessageType(LOG_NONE);
}
rc = YesNoBox('i', "Added %d projects to bulk file %s. Do you want to submit this to EDGAR now?", num_files, bulk);
if(rc == IDYES){
hES = EDGARSessionOpenSession(0);
if (IsError(hES)) {
rc = GetLastError();
s1 = GetLastErrorMessage();
MessageBox('x', "Error %08X\r\r%s", rc, s1);
return ERROR_EXIT;
}
if(IsValidHandle(hES)==false){
MessageBox('x',"Cannot get handle to EDGAR Session.");
return ERROR_EXIT;
}
After our script has finished creating our file, we can get started on actually submitting it to the SEC. First, we need to ask the user if he or she actually wants to submit this filing. If the user hits Yes, we can go ahead and begin. We open a connection to the SEC by using the EDGARSessionOpenSession function. This will use the default information in the Preferences section of GoFiler to log into the SEC . If we got an error instead of the correct handle or if the handle is not a valid handle, we can return an error message and then return with an error, since we cannot file it to the SEC.
eProps = EDGARSessionGetProperties(hES);
s1 = FormatString("servlet/MultiHandlerServlet?session=%s&loginCIK=%s&handler=uploadfile&ftype=%s",
eProps["SessionID"],
eProps["LoginCIK"],
bulkType);
params["FileData"] = "%%FileData%%";
hHTTP = EDGARSessionGetHTTPHandle(hES);
response = HTTPPost(hHTTP,s1,params,bulk);
rc = GetLastError();
AddMessage(log,"Sending Request to EDGAR: %s",s1);
Once we have our EDGAR connection opened, we can use the EDGARSessionGetPropreties function to get some attributes about our session we’ll need to build our connection string. The variable s1 will hold our actual POST request. I designed this string by previously filing a test filing to the SEC and copying the URL to which that request was sent. Then I replaced the Session, LoginCIK, and ftype variables in the string with the real values from the EDGARSessionGetProperties function and the bulk file type we created earlier. We also need to create an array called params, containing a single key/value pair with “FileData” as the key and “%%FileData%%” as the value. The documentation for the HTTPPost function goes into more detail why, but the short version is that it’s required to attach a file to a POST in Legato. We can then use our HTTPPost function to actually submit our crafted request to the EDGAR System and get the result back with the GetLastError function.
if(IsError(rc)){
MessageBox('x',"Cannot connect to SEC, error code %0x",rc);
}
else{
if(FindInString(response,"Discontinued")>0){
MessageBox('x',"EDGAR Session Discontinued");
}
else{
accession_pos = 0;
for(ix=0; ix < size; ix++){
accession_pos = FindInString(response,ACCESSION_START,accession_pos);
if(accession_pos < 0){
response_out = GetFilePath(bulk);
response_out = AddPaths(response_out, "Response.html");
StringToFile(response,response_out);
MessageBox('x',"An unknown error has occurred. See filed %s for details.",response_out);
LogDisplay(log);
return ERROR_NONE;
}
accession_num = GetStringSegment(response,accession_pos+GetStringLength(ACCESSION_START),20);
accession_pos+= GetStringLength(ACCESSION_START)+20;
csv_response[ix][0] = files[ix];
csv_response[ix][1] = accession_num;
}
response_out = GetFilePath(bulk);
response_out = AddPaths(response_out, "Accession Numbers.csv");
CSVWriteTable(csv_response,response_out);
MessageBox('x',"File Submitted Successfully. See file %s for details.",response_out);
}
}
}
LogDisplay(log);
return ERROR_NONE;
}
Now we should take a look at the return code we received. If the code is an error, like a 404 or 503 or something similar, we can display a message to the user to let him or her know we couldn’t submit the filing. If we received no error, then the communication was successful, so we can look at the response from the SEC. Note that this does not necessarily mean the filing was submitted. If the response contained the word “Discontinued”, we can assume it was a “Session Discontinued” error, so the file was probably not submitted successfully, and the user needs to be made aware of it. Otherwise, we can try to parse our response to match accession numbers to file names.
For each file we submitted, we can run the FindInString function to find where the accession number should start. If we don’t find an accession number for each file, then it means we either have no accession numbers, and an unknown error has occurred that we couldn’t detect before, or we don’t have the same number of accession numbers as files, which is also a problem. In either case, we can return an error message to the user, save the response from the SEC to a folder, and then exit.
Otherwise, for each accession number we find, we can get the accession number after locating each one with the GetStringSegment function and set the value into a table containing the file names in column 0 and the accession numbers in column 1. After we’ve processed each file, we can use the CSVWriteTable function to write out our table in a nice, easy-to-read CSV format, display to the user a success message, display the log, and exit without an error.
So this completes our discussion of how to create and submit a bulk filing to EDGAR. There are, of course, many enhancements that can be made to this script to tailor it to a specific situation or use case. Here’s a complete version of this script, with all changes included in it.
//
// Function Definitions and Globals
// --------------------------------
#define ACCESSION_START "<BR> Accession Number: "
#define TEST_DIRECTIVE "%%%TEST%%%"
#define NAME_DIRECTIVE "%%%NAME%%%"
#define OPEN "<?xml version=\"1.0\" ?>\r\n<bul:edgarBulkSubmission xmlns:bul=\"http://www.sec.gov/edgar/bulkfiling\">"
#define LIVE_FLAG "<bul:liveTestFlag>"+TEST_DIRECTIVE+"</bul:liveTestFlag>"
#define ATTACHMENTS_OPEN "<bul:attachments>"
#define ATTACHMENTS_CLOSE "\r\n</bul:attachments>"
#define ATTACHMENT_OPEN "\r\n<bul:attachment>\r\n<bul:submissionName>" + NAME_DIRECTIVE + "</bul:submissionName>\r\n<bul:contents>\r\n"
#define ATTACHMENT_CLOSE "</bul:contents>\r\n</bul:attachment>"
#define CLOSE "</bul:edgarBulkSubmission>"
#define SEC_FILE_LIMIT 209715200
int bd_load ();
void bd_set_state ();
int bd_action (int id, int action);
int bd_validate ();
string files[];
string bulk;
string live;
int run(int id, string mode);
void setup();
void main(){
setup();
if(GetScriptParent()=="LegatoIDE"){
run(0,"preprocess");
}
}
void setup(){
string menu[];
menu["Code"] = "CREATE_BULK_FILING";
menu["MenuText"] = "Create Bulk Filing";
menu["Description"] = "Compresses multiple GFP files into a single BULK XML File.";
MenuAddFunction(menu);
MenuSetHook("CREATE_BULK_FILING",GetScriptFilename(),"run");
}
int run(int id, string mode) {
int size;
int bytes;
int num_files;
int ix;
int rc;
int limit;
int accession_pos;
handle hES;
handle hHTTP;
handle hResult;
handle log;
handle hAttachment;
handle hEDAC;
string bulkType;
string fname;
string enc_fname;
string outfile;
string params[];
string test;
string file;
string attachment;
string eProps[];
string session;
string s1;
string response;
string csv_response[][];
string accession_num;
string response_out;
if (mode != "preprocess") {
return ERROR_NONE;
}
// Load Settings
bulk = GetSetting("settings","bulk");
files = ExplodeString(GetSetting("settings","files"),"|");
// Run Dialog
rc = DialogBox("BulkDialog", "bd_");
if (IsError(rc)) {
return rc;
}
// Set up
log = LogCreate("Create Bulk Filing");
hResult = CreateFile(bulk);
if (IsError(hResult)) {
MessageBox('x', "Could not save to file. %s", TranslateWindowsError(GetLastError()));
return ERROR_FILE;
}
ProgressOpen("Creating Bulk Filing...", 0);
// Start of File
hAttachment = PoolCreate();
WriteLine(hResult, OPEN);
test = ReplaceInString(LIVE_FLAG, TEST_DIRECTIVE, live);
bulkType = "BulkTest";
if (live == "LIVE"){
bulkType = "BulkLive";
}
WriteLine(hResult, test);
// Write Attachments
limit = -1;
size = ArrayGetAxisDepth(files);
WriteBlock(hResult, ATTACHMENTS_OPEN, GetStringLength(ATTACHMENTS_OPEN));
for (ix = 0; ix < size; ix++) {
ProgressUpdate(ix, size);
ProgressSetStatus("File %d of %d", ix + 1, size);
PoolReset(hAttachment);
fname = files[ix];
outfile = AddPaths(GetTempFileFolder(), GetFilename(fname));
outfile = ClipFileExtension(outfile);
outfile = outfile + ".xml";
AddMessage(log, "Adding file %s to bulk filing", fname);
hEDAC = EDACOpenFile(fname);
if (IsError(hEDAC)) {
rc = GetLastError();
LogSetMessageType(LOG_ERROR);
AddMessage(log, " Cannot open filing, error 0x%08x", rc);
LogSetMessageType(LOG_NONE);
continue;
}
rc = EDACSaveFile(hEDAC, outfile);
CloseHandle(hEDAC);
if (IsError(rc)) {
LogSetMessageType(LOG_ERROR);
AddMessage(log, " Cannot save filing, error 0x%08x", rc);
LogSetMessageType(LOG_NONE);
continue;
}
file = FileToString(outfile);
rc = GetLastError();
if (IsError(rc)) {
LogSetMessageType(LOG_ERROR);
AddMessage(log, " Cannot add file to filing, error 0x%08x", rc);
LogSetMessageType(LOG_NONE);
continue;
}
// Build Attachment Code
file = EncodeString(file);
PoolAppend(hAttachment, ReplaceInString(ATTACHMENT_OPEN, NAME_DIRECTIVE, GetFilename(outfile)));
PoolAppend(hAttachment, file);
PoolAppend(hAttachment, ATTACHMENT_CLOSE);
// Write to File
attachment = PoolGetPool(hAttachment);
WriteBlock(hResult, attachment, PoolGetPosition(hAttachment));
DeleteFile(outfile);
bytes = GetFilePosition(hResult);
if (bytes >= SEC_FILE_LIMIT) {
limit = ix;
}
num_files++;
}
// Finish File
WriteLine(hResult, ATTACHMENTS_CLOSE);
WriteLine(hResult, CLOSE);
CloseHandle(hResult);
ProgressClose();
if (limit >= 0) {
LogSetMessageType(LOG_WARNING);
AddMessage(log, " Bulk filing size (%d) exceeds EDGAR transmission (%d) limit.", GetFileSize(hResult), SEC_FILE_LIMIT);
AddMessage(log, " Remove files after entry %d (%s).", limit + 1, files[limit]);
LogSetMessageType(LOG_NONE);
}
rc = YesNoBox('i', "Added %d projects to bulk file %s. Do you want to submit this to EDGAR now?", num_files, bulk);
if(rc == IDYES){
hES = EDGARSessionOpenSession(0);
if (IsError(hES)) {
rc = GetLastError();
s1 = GetLastErrorMessage();
MessageBox('x', "Error %08X\r\r%s", rc, s1);
return ERROR_EXIT;
}
if(IsValidHandle(hES)==false){
MessageBox('x',"Cannot get handle to EDGAR Session.");
return ERROR_EXIT;
}
eProps = EDGARSessionGetProperties(hES);
s1 = FormatString("servlet/MultiHandlerServlet?session=%s&loginCIK=%s&handler=uploadfile&ftype=%s",
eProps["SessionID"],
eProps["LoginCIK"],
bulkType);
params["FileData"] = "%%FileData%%";
hHTTP = EDGARSessionGetHTTPHandle(hES);
response = HTTPPost(hHTTP,s1,params,bulk);
rc = GetLastError();
AddMessage(log,"Sending Request to EDGAR: %s",s1);
if(IsError(rc)){
MessageBox('x',"Cannot connect to SEC, error code %0x",rc);
}
else{
if(FindInString(response,"Discontinued")>0){
MessageBox('x',"EDGAR Session Discontinued");
}
else{
accession_pos = 0;
for(ix=0; ix < size; ix++){
accession_pos = FindInString(response,ACCESSION_START,accession_pos);
if(accession_pos < 0){
response_out = GetFilePath(bulk);
response_out = AddPaths(response_out, "Response.html");
StringToFile(response,response_out);
MessageBox('x',"An unknown error has occurred. See filed %s for details.",response_out);
LogDisplay(log);
return ERROR_NONE;
}
accession_num = GetStringSegment(response,accession_pos+GetStringLength(ACCESSION_START),20);
accession_pos+= GetStringLength(ACCESSION_START)+20;
csv_response[ix][0] = files[ix];
csv_response[ix][1] = accession_num;
}
response_out = GetFilePath(bulk);
response_out = AddPaths(response_out, "Accession Numbers.csv");
CSVWriteTable(csv_response,response_out);
MessageBox('x',"File Submitted Successfully. See file %s for details.",response_out);
}
}
}
LogDisplay(log);
return ERROR_NONE;
}
//
// Dialog and Defines
// --------------------------------
#define DC_BULK 201
#define DC_LIST 301
#define DC_ADD 101
#define DC_ADDFOLDER 102
#define DC_REMOVE 103
#define DC_BROWSE 104
#define DC_LIVE 105
#define DC_CLEAR 106
#define ADD_FILTER "All Projects|*.gfp;*.eis;*.xml;&GoFiler Projects|*.gfp&EDGAR Internet Submissions|*.eis&XML Submission Files|*.xml&All Files *.*|*.*"
#define SAVE_FILTER "XML Submission Files|*.xml&All Files *.*|*.*"
int bd_load() {
int ix, mx;
// Load List
mx = ArrayGetAxisDepth(files);
for (ix = 0; ix < mx; ix++) {
ListBoxAddItem(DC_LIST, files[ix]);
}
ControlSendMessage(DC_LIST, LB_SETHORIZONTALEXTENT, 800, 0);
EditSetText(DC_BULK, bulk);
bd_set_state();
return ERROR_NONE;
}
void bd_set_state() {
int fx;
int ix;
fx = ListBoxGetSelectIndex(DC_LIST);
ix = ListBoxGetItemCount(DC_LIST);
if (ix > 0) {
ControlEnable(DC_CLEAR);
}
else {
ControlDisable(DC_CLEAR);
}
if (fx >= 0) {
ControlEnable(DC_REMOVE);
}
else {
ControlDisable(DC_REMOVE);
}
}
int bd_action(int id, int action) {
string tba[];
string path;
string ext;
int rc, ix, mx;
switch (id) {
case DC_ADD:
tba = BrowseOpenFiles("Select Projects", ADD_FILTER);
if (GetLastError() == ERROR_CANCEL) {
break;
}
mx = ArrayGetAxisDepth(tba);
for (ix = 0; ix < mx; ix++) {
rc = ListBoxFindItem(DC_LIST, tba[ix]);
if (IsError(rc)) {
ListBoxAddItem(DC_LIST, tba[ix]);
}
}
bd_set_state();
break;
case DC_ADDFOLDER:
path = BrowseFolder("Select Projects");
if (GetLastError() == ERROR_CANCEL) {
break;
}
tba = EnumerateFiles(path, FOLDER_LOAD_NO_FOLDER_NAV | FOLDER_LOAD_RECURSE);
mx = ArrayGetAxisDepth(tba);
for (ix = 0; ix < mx; ix++) {
ext = MakeLowerCase(GetExtension(tba[ix]));
if ((ext == ".gfp") || (ext == ".eis")) {
tba[ix] = AddPaths(path, tba[ix]);
rc = ListBoxFindItem(DC_LIST, tba[ix]);
if (IsError(rc)) {
ListBoxAddItem(DC_LIST, tba[ix]);
}
}
}
bd_set_state();
break;
case DC_REMOVE:
rc = ListBoxGetSelectIndex(DC_LIST);
if (rc >= 0) {
ListBoxDeleteItem(DC_LIST, rc);
while (rc >= ListBoxGetItemCount(DC_LIST)) {
rc--;
}
if (rc >= 0) {
ListBoxSetSelectIndex(DC_LIST, rc);
}
}
bd_set_state();
break;
case DC_CLEAR:
ListBoxReset(DC_LIST);
bd_set_state();
break;
case DC_BROWSE:
path = EditGetText(DC_BULK);
path = BrowseSaveFile("Save Bulk Submission", SAVE_FILTER, path);
if (GetLastError() == ERROR_CANCEL) {
break;
}
EditSetText(DC_BULK, path);
break;
case DC_LIST:
if (action == LBN_SELCHANGE) {
bd_set_state();
}
break;
}
return ERROR_NONE;
}
int bd_validate() {
string tba[];
string name;
string s1;
int ix;
int size;
// Check List
tba = ListBoxGetArray(DC_LIST);
if (ArrayGetAxisDepth(tba) == 0) {
MessageBox('x', "Submission must have at least one file.");
return ERROR_SOFT | DC_LIST;
}
// Check Name
name = EditGetText(DC_BULK);
if (name == "") {
MessageBox('x', "Bulk Submission File is a required field.");
return ERROR_SOFT | DC_LIST;
}
if (MakeLowerCase(GetExtension(name)) != ".xml") {
MessageBox('x', "Bulk Submission File must be an XML file");
return ERROR_SOFT | DC_LIST;
}
if (CanAccessFile(name) == false){
MessageBox('x', "Bulk Submission File must be a valid file location and not open by other applications.");
return ERROR_SOFT | DC_LIST;
}
// Save Values
bulk = name;
files = tba;
if (CheckboxGetState(DC_LIVE) == BST_CHECKED){
live = "LIVE";
}
else{
live = "TEST";
}
s1 = ImplodeArray(files, "|");
PutSetting("settings", "bulk", bulk);
PutSetting("settings", "files", s1);
return ERROR_NONE;
}
#beginresource
BulkDialog DIALOGEX 0, 0, 344, 190
EXSTYLE WS_EX_DLGMODALFRAME
STYLE DS_MODALFRAME | DS_3DLOOK | WS_POPUP | WS_VISIBLE | WS_CAPTION | WS_SYSMENU
CAPTION "Bulk Filing"
FONT 8, "MS Shell Dlg", 400, 0
{
CONTROL "Files", -1, "static", SS_LEFT | WS_CHILD | WS_VISIBLE, 6, 4, 20, 8, 0
CONTROL "", -1, "static", SS_ETCHEDFRAME | WS_CHILD | WS_VISIBLE, 26, 9, 314, 1, 0
CONTROL "", DC_LIST, "listbox", LBS_NOTIFY | WS_CHILD | WS_VISIBLE | WS_BORDER | WS_VSCROLL | WS_HSCROLL | WS_TABSTOP, 12, 16, 276, 120, 0
CONTROL "&Add...", DC_ADD, "button", BS_PUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 292, 16, 45, 12, 0
CONTROL "Add &Folder...", DC_ADDFOLDER, "button", BS_PUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 292, 31, 45, 12, 0
CONTROL "&Remove", DC_REMOVE, "button", BS_PUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 292, 46, 45, 12, 0
CONTROL "&Clear", DC_CLEAR, "button", BS_PUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 292, 61, 45, 12, 0
CONTROL "&LIVE", DC_LIVE, "button", BS_AUTOCHECKBOX | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 296, 76, 45, 12, 0
CONTROL "Submission", -1, "static", SS_LEFT | WS_CHILD | WS_VISIBLE, 6, 137, 42, 8, 0
CONTROL "", -1, "static", SS_ETCHEDFRAME | WS_CHILD | WS_VISIBLE, 48, 142, 292, 1, 0
CONTROL "&Bulk Submission File:", -1, "static", SS_LEFT | WS_CHILD | WS_VISIBLE, 12, 150, 72, 8, 0
CONTROL "", DC_BULK, "edit", ES_AUTOHSCROLL | WS_CHILD | WS_VISIBLE | WS_BORDER | WS_TABSTOP, 88, 148, 200, 12
CONTROL "Browse...", DC_BROWSE, "button", BS_PUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 292, 148, 45, 12, 0
CONTROL "", -1, "static", SS_ETCHEDFRAME | WS_CHILD | WS_VISIBLE, 6, 166, 335, 1, 0
CONTROL "&Create", IDOK, "BUTTON", BS_DEFPUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 232, 172, 50, 14
CONTROL "Cancel", IDCANCEL, "BUTTON", BS_PUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 287, 172, 50, 14
}
#endresource
 Steven Horowitz has been working for Novaworks for over five years as a technical expert with a focus on EDGAR HTML and XBRL. Since the creation of the Legato language in 2015, Steven has been developing scripts to improve the GoFiler user experience. He is currently working toward a Bachelor of Sciences in Software Engineering at RIT and MCC. |
Additional Resources
Legato Script Developers LinkedIn Group
Primer: An Introduction to Legato
Quicksearch
Categories
Calendar
Feeds
