Last week, Dave went over how to build a basic UI for our Bulk Filing function. This week, we’re expanding that a little bit by adding a “clear all” function to drop all items from the list, a “LIVE” checkbox to mark our submission as a live filing, and a setting to store the previously chosen files to reload if we need to rebuild the filing.
Friday, December 21. 2018
LDC #115: Bulk Filings Part 2
We’re also going to add the logic for converting the project files into XML files, which is the required format to create a bulk filing submission. The RunMenuFunction function will be used to access the File Open operation to open the file, and then again to do a Save As operation, and finally once more to do the Close operation. For each project we attach, GoFiler will quickly open it, save it, and then close it. This does mean that it will take a second or so for each filing to be processed, so be careful not to add to many files to a single bulk filing, or you could be sitting there for 20-30 minutes if you have a couple thousand files! It’s also important to remember that GoFiler does a background lookup on the CIK/CCC in the filing every time it opens a project, which means that you’ll be pinging the SEC once for each item in the bulk filing as GoFiler creates the submission.
The last part of the script is the logic to create the bulk filing itself. Once you have the XML files created, it’s fairly trivial to actually put them into a bulk submission, especially since our primary execution loop already iterates over all of them anyway. So let’s take a look at some of our new defines:
// // Function Definitions and Globals // -------------------------------- #define TEST_DIRECTIVE "%%%TEST%%%" #define NAME_DIRECTIVE "%%%NAME%%%" #define OPEN "<?xml version=\"1.0\" ?>\r\n<bul:edgarBulkSubmission xmlns:bul=\"http://www.sec.gov/edgar/bulkfiling\">" #define LIVE_FLAG "<bul:liveTestFlag>"+TEST_DIRECTIVE+"</bul:liveTestFlag>" #define ATTACHMENTS_OPEN "<bul:attachments>\r\n" #define ATTACHMENTS_CLOSE "\r\n</bul:attachments>" #define ATTACHMENT_OPEN "<bul:attachment>\r\n<bul:submissionName>\r\n"+NAME_DIRECTIVE+"\r\n</bul:submissionName>\r\n<bul:contents>\r\n" #define ATTACHMENT_CLOSE "</bul:contents>\r\n</bul:attachment>" #define CLOSE "</bul:edgarBulkSubmission>"
These defines are the basic structure of our bulk filing. OPEN is the opening XML tag for the bulk submission. LIVE_FLAG is the test or live flag that needs to be set. ATTACHMENTS_OPEN is the open tag of the XML wrapper tag that goes around all attached files. ATTACHMENTS_CLOSE is the corresponding closing tag. ATTACHMENT_OPEN is the opening of the tag that is used for each individual file attached to the submission. ATTACHMENT_CLOSE is the close tag for each attachment. CLOSE is the final closing XML tag in the submission file. For each of these, we have “directive” defines, which are going to be replaced in the string with ReplaceInString operations later. These directives are just simple placeholders, which will have actual data put in later by functions in the script.
int bd_load (); void bd_set_state (); int bd_action (int id, int action); int bd_validate (); string files[]; string bulk; string live; int run(int id, string mode); void setup();
We’re also adding the global variables live, which store the live/test flag status. The functions run and setup are also added. The function run is the primary action function and where most of our actual work takes place, and setup is just a normal function to add the function to the toolbar, so we won’t need to discuss it in depth.
int run(int id, string mode) {
... omitted variable declarations ...
if(mode!="preprocess"){
return ERROR_NONE;
}
bulk = GetSetting("settings","bulk");
files = ExplodeString(GetSetting("settings","files"),"|");
log = LogCreate("Create Bulk Filing");
rc = DialogBox("BulkDialog", "bd_");
if (rc == ERROR_NONE){
pool = PoolCreate();
hAttachment = PoolCreate();
PoolAppend(pool,OPEN);
PoolAppendNewLine(pool);
test = ReplaceInString(LIVE_FLAG,TEST_DIRECTIVE,live);
PoolAppend(pool,test);
PoolAppendNewLine(pool);
Our run function starts off by setting our bulk and files variables, by using the GetSetting function to recall the values from the last time the function was run. Next, we check to make sure we’re running in preprocess mode, or the script just exits. If we’re in preprocess mode, we can continue, and create our log file, and enter into our dialog control, which Dave wrote last week. It’s got a few minor changes we’ll go over after we talk about run, but it’s mostly the same basic dialog control. The Dialog is going to set our bulk variable, our live variable, and our files variable, and after it closes, if the response code was ERROR_NONE, we can proceed with creating our bulk filing. I decided to use a pair of String Pools for this, pool and hAttachment. The pool variable represents the entire filing, and hAttachment is going to represent a single file within the attachment. String Pools are useful because they are much faster when adding chunks of data to them, since they do not need to reallocate space every single time data is added to them. We can start filling up our pool variable by appending the OPEN define, creating the live/test flag by using ReplaceInString, and then appending that as well.
size = ArrayGetAxisDepth(files);
for(ix=0;ix<size;ix++){
attachment = "";
fname = files[ix];
AddMessage(log,"Adding file %s to bulk filing",fname);
RunMenuFunction("FILE_OPEN","Filename:"+fname);
outfile = ClipFileExtension(fname);
outfile = outfile+".xml";
params = "Filename:"+outfile+";Query:FALSE";
rc = RunMenuFunction("FILE_SAVE_AS",params);
LogSetMessageType(LOG_ERROR);
if(IsError(rc)){
RunMenuFunction("FILE_CLOSE");
AddMessage(log," Cannot add file to filing, error %0x",rc);
continue;
}
RunMenuFunction("FILE_CLOSE");
For each file, we can go ahead and get the file name, log that we’re processing the file, and use RunMenuFunction to open it. Once it’s open, we can get the file name, create an XML filename using ClipFileExtension and appending “.xml” to the filename, and then use RunMenuFunction to save it as XML. If this process returns an error, we can log that it was unable to be saved, and then continue on to the next file. If there was no error, we can just use RunMenuFunction to close the file.
file = FileToString(outfile);
rc = GetLastError();
if(IsError(rc)){
AddMessage(log," Cannot add file to filing, error %0x",rc);
continue;
}
num_files ++;
LogSetMessageType(LOG_NONE);
file = EncodeString(file);
PoolAppend(hAttachment,ReplaceInString(ATTACHMENT_OPEN,NAME_DIRECTIVE,GetFilename(outfile)));
PoolAppend(hAttachment,file);
PoolAppend(hAttachment,ATTACHMENT_CLOSE);
DeleteFile(outfile);
}
Now that our file is an XML file, we can use FileToString to read it. We need to test to make sure we actually read it, if there was an error then we can log it and continue onto the next file. Otherwise, we can increment the number of processed files, encode the XML file using EncodeString, and use ReplaceInString to fill in the name parameter of our ATTACHMENT_OPEN string. We can then use PoolAppend to add our attachment open tag, our attachment contents, and our attachment close tag, before deleting the XML file we created as a cleanup operation.
attachment = PoolGetPool(hAttachment);
PoolAppend(pool,ATTACHMENTS_OPEN);
PoolAppend(pool,attachment);
PoolAppend(pool,ATTACHMENTS_CLOSE);
PoolAppendNewLine(pool);
PoolAppend(pool,CLOSE);
PoolWriteFile(pool,bulk);
LogDisplay(log);
MessageBox('i',"Added %d projects to bulk file %s.",num_files, bulk);
}
return 0;
}
After we’ve processed every file, we can go ahead and get our entire hAttachment pool as a single string, and then use PoolAppend again to add our open tag for attachments, then our actual attachment contents, then the closing tag. Finally, we can append a new line, add our close tag, and write the pool out to our output XML file. We will also display the log to the user and show the user a message box to say how many files were attached successfully.
That finishes our primary run loop. Now let’s take a look at a few of our modified dialog controls, to make using this a little easier.
The load function for the dialog is the same as last week, so we won’t talk about it here.
void bd_set_state() {
int fx;
int ix;
fx = ListBoxGetSelectIndex(DC_LIST);
ix = ListBoxGetItemCount(DC_LIST);
if (ix > 0) {
ControlEnable(DC_CLEAR);
}
else {
ControlDisable(DC_CLEAR);
}
if (fx >= 0) {
ControlEnable(DC_REMOVE);
}
else {
ControlDisable(DC_REMOVE);
}
}
The bd_set_state function is slightly different. The “Clear” button was added to the dialog, so we need to enable or disable it based on the number of items in the list of files. If the list of files is greater than zero, we can enable the button. If it’s not, then we can disable it. Other functionality in here is unchanged.
int bd_action(int id, int action) {
string tba[];
string path;
int rc, ix, mx;
switch (id) {
... code omitted ...
case DC_CLEAR:
ListBoxReset(DC_LIST);
bd_set_state();
break;
case DC_BROWSE:
.... code omitted ...
The bd_action function is also slightly modified. We had to add a new case to the main switch statement for the DC_CLEAR button. If the button is pressed, we simply reset the list, trigger the set state function, and then break. Everything else is unchanged.
int bd_validate() {
... omitted code ...
bulk = name;
files = tba;
if (CheckboxGetState(DC_LIVE) == BST_CHECKED){
live = "LIVE";
}
else{
live = "TEST";
}
s1 = ImplodeArray(files,"|");
PutSetting("settings","bulk",bulk);
PutSetting("settings","files",s1);
return ERROR_NONE;
}
The last function we’ve changed, bd_validate, needs some extra logic at the end of it. When we’re storing the variables for later use, we also need to check the new checkbox DC_LIVE. If it’s checked, our live variable is set to “LIVE”. Otherwise, it’s set to “TEST”. We also need to use the PutSetting function to store our bulk and files variables as settings, so we can restore them when the user runs the dialog again. The files variable is an array, so we can’t just store it. Because of that I’m using the ImplodeArray function to convert it into a string. I’ve chosen to use the “|” character as a delimiter, because it’s not a valid character in a filename, so when we explode the string later it won’t cause any problems.
This script can definitely be further modified, but I think this is a good starting point. It could have additional features, like selecting a directory and having it add all files in a directory. It could also use HTTPPost functions to actually log into the SEC’s EDGAR system and submit the filing, and parse the response back to give a mapping of accession numbers to files. While features like this would be nice, it would also greatly increase the complexity of the script, and as always we must be mindful of spending a lot of time for relatively little gain.
Here is the complete script without commentary:
//
// Function Definitions and Globals
// --------------------------------
#define TEST_DIRECTIVE "%%%TEST%%%"
#define NAME_DIRECTIVE "%%%NAME%%%"
#define OPEN "<?xml version=\"1.0\" ?>\r\n<bul:edgarBulkSubmission xmlns:bul=\"http://www.sec.gov/edgar/bulkfiling\">"
#define LIVE_FLAG "<bul:liveTestFlag>"+TEST_DIRECTIVE+"</bul:liveTestFlag>"
#define ATTACHMENTS_OPEN "<bul:attachments>\r\n"
#define ATTACHMENTS_CLOSE "\r\n</bul:attachments>"
#define ATTACHMENT_OPEN "<bul:attachment>\r\n<bul:submissionName>\r\n"+NAME_DIRECTIVE+"\r\n</bul:submissionName>\r\n<bul:contents>\r\n"
#define ATTACHMENT_CLOSE "</bul:contents>\r\n</bul:attachment>"
#define CLOSE "</bul:edgarBulkSubmission>"
int bd_load ();
void bd_set_state ();
int bd_action (int id, int action);
int bd_validate ();
string files[];
string bulk;
string live;
int run(int id, string mode);
void setup();
void main(){
setup();
}
void setup(){
string menu[];
menu["Code"] = "CREATE_BULK_FILING";
menu["MenuText"] = "Create Bulk Filing";
menu["Description"] = "Compresses multiple GFP files into a single BULK XML File.";
MenuAddFunction(menu);
MenuSetHook("CREATE_BULK_FILING",GetScriptFilename(),"run");
}
int run(int id, string mode) {
int size;
int num_files;
int ix;
int rc;
handle pool;
handle log;
handle hAttachment;
string fname;
string enc_fname;
string outfile;
string params;
string test;
string file;
string attachment;
if(mode!="preprocess"){
return ERROR_NONE;
}
bulk = GetSetting("settings","bulk");
files = ExplodeString(GetSetting("settings","files"),"|");
log = LogCreate("Create Bulk Filing");
rc = DialogBox("BulkDialog", "bd_");
if (rc == ERROR_NONE){
pool = PoolCreate();
hAttachment = PoolCreate();
PoolAppend(pool,OPEN);
PoolAppendNewLine(pool);
test = ReplaceInString(LIVE_FLAG,TEST_DIRECTIVE,live);
PoolAppend(pool,test);
PoolAppendNewLine(pool);
size = ArrayGetAxisDepth(files);
for(ix=0;ix<size;ix++){
attachment = "";
fname = files[ix];
AddMessage(log,"Adding file %s to bulk filing",fname);
RunMenuFunction("FILE_OPEN","Filename:"+fname);
outfile = ClipFileExtension(fname);
outfile = outfile+".xml";
params = "Filename:"+outfile+";Query:FALSE";
rc = RunMenuFunction("FILE_SAVE_AS",params);
LogSetMessageType(LOG_ERROR);
if(IsError(rc)){
RunMenuFunction("FILE_CLOSE");
AddMessage(log," Cannot add file to filing, error %0x",rc);
continue;
}
RunMenuFunction("FILE_CLOSE");
file = FileToString(outfile);
rc = GetLastError();
if(IsError(rc)){
AddMessage(log," Cannot add file to filing, error %0x",rc);
continue;
}
num_files ++;
LogSetMessageType(LOG_NONE);
file = EncodeString(file);
PoolAppend(hAttachment,ReplaceInString(ATTACHMENT_OPEN,NAME_DIRECTIVE,GetFilename(outfile)));
PoolAppend(hAttachment,file);
PoolAppend(hAttachment,ATTACHMENT_CLOSE);
DeleteFile(outfile);
}
attachment = PoolGetPool(hAttachment);
PoolAppend(pool,ATTACHMENTS_OPEN);
PoolAppend(pool,attachment);
PoolAppend(pool,ATTACHMENTS_CLOSE);
PoolAppendNewLine(pool);
PoolAppend(pool,CLOSE);
PoolWriteFile(pool,bulk);
LogDisplay(log);
MessageBox('i',"Added %d projects to bulk file %s.",num_files, bulk);
}
return 0;
}
//
// Dialog and Defines
// --------------------------------
#define DC_BULK 201
#define DC_LIST 301
#define DC_ADD 101
#define DC_REMOVE 102
#define DC_BROWSE 103
#define DC_LIVE 104
#define DC_CLEAR 105
#define ADD_FILTER "All Projects|*.gfp;*.eis;*.xml;&GoFiler Projects|*.gfp&EDGAR Internet Submissions|*.eis&XML Submission Files|*.xml&All Files *.*|*.*"
#define SAVE_FILTER "XML Submission Files|*.xml&All Files *.*|*.*"
int bd_load() {
int ix, mx;
// Load List
mx = ArrayGetAxisDepth(files);
for (ix = 0; ix < mx; ix++) {
ListBoxAddItem(DC_LIST, files[ix]);
}
EditSetText(DC_BULK, bulk);
bd_set_state();
return ERROR_NONE;
}
void bd_set_state() {
int fx;
int ix;
fx = ListBoxGetSelectIndex(DC_LIST);
ix = ListBoxGetItemCount(DC_LIST);
if (ix > 0) {
ControlEnable(DC_CLEAR);
}
else {
ControlDisable(DC_CLEAR);
}
if (fx >= 0) {
ControlEnable(DC_REMOVE);
}
else {
ControlDisable(DC_REMOVE);
}
}
int bd_action(int id, int action) {
string tba[];
string path;
int rc, ix, mx;
switch (id) {
case DC_ADD:
tba = BrowseOpenFiles("Select Projects", ADD_FILTER);
if (GetLastError() == ERROR_CANCEL) {
break;
}
mx = ArrayGetAxisDepth(tba);
for (ix = 0; ix < mx; ix++) {
rc = ListBoxFindItem(DC_LIST, tba[ix]);
if (IsError(rc)) {
ListBoxAddItem(DC_LIST, tba[ix]);
}
}
bd_set_state();
break;
case DC_REMOVE:
rc = ListBoxGetSelectIndex(DC_LIST);
if (rc >= 0) {
ListBoxDeleteItem(DC_LIST, rc);
while (rc >= ListBoxGetItemCount(DC_LIST)) {
rc--;
}
if (rc >= 0) {
ListBoxSetSelectIndex(DC_LIST, rc);
}
}
bd_set_state();
break;
case DC_CLEAR:
ListBoxReset(DC_LIST);
bd_set_state();
break;
case DC_BROWSE:
path = EditGetText(DC_BULK);
path = BrowseSaveFile("Save Bulk Submission", SAVE_FILTER, path);
if (GetLastError() == ERROR_CANCEL) {
break;
}
EditSetText(DC_BULK, path);
break;
case DC_LIST:
if (action == LBN_SELCHANGE) {
bd_set_state();
}
break;
}
return ERROR_NONE;
}
int bd_validate() {
string tba[];
string name;
string s1;
int ix;
int size;
// Check List
tba = ListBoxGetArray(DC_LIST);
if (ArrayGetAxisDepth(tba) == 0) {
MessageBox('x', "Submission must have at least one file.");
return ERROR_SOFT | DC_LIST;
}
// Check Name
name = EditGetText(DC_BULK);
if(MakeLowerCase(GetExtension(name))!=".xml"){
MessageBox('x',"Bulk Submission File must be an XML file");
return ERROR_SOFT | DC_LIST;
}
if (name == "") {
MessageBox('x', "Bulk Submission File is a required field.");
return ERROR_SOFT | DC_LIST;
}
if(CanAccessFile(name)==false){
MessageBox('x', "Bulk Submission File must be a valid file location.");
return ERROR_SOFT | DC_LIST;
}
// Save Values
bulk = name;
files = tba;
if (CheckboxGetState(DC_LIVE) == BST_CHECKED){
live = "LIVE";
}
else{
live = "TEST";
}
s1 = ImplodeArray(files,"|");
PutSetting("settings","bulk",bulk);
PutSetting("settings","files",s1);
return ERROR_NONE;
}
#beginresource
BulkDialog DIALOGEX 0, 0, 344, 190
EXSTYLE WS_EX_DLGMODALFRAME
STYLE DS_MODALFRAME | DS_3DLOOK | WS_POPUP | WS_VISIBLE | WS_CAPTION | WS_SYSMENU
CAPTION "Bulk Filing"
FONT 8, "MS Shell Dlg", 400, 0
{
CONTROL "Files", -1, "static", SS_LEFT | WS_CHILD | WS_VISIBLE, 6, 4, 20, 8, 0
CONTROL "", -1, "static", SS_ETCHEDFRAME | WS_CHILD | WS_VISIBLE, 26, 9, 314, 1, 0
CONTROL "", DC_LIST, "listbox", LBS_NOTIFY | WS_CHILD | WS_VISIBLE | WS_BORDER | WS_VSCROLL | WS_TABSTOP, 12, 16, 276, 120, 0
CONTROL "&Add...", DC_ADD, "button", BS_PUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 292, 16, 45, 12, 0
CONTROL "&Remove", DC_REMOVE, "button", BS_PUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 292, 31, 45, 12, 0
CONTROL "&Clear", DC_CLEAR, "button", BS_PUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 292, 46, 45, 12, 0
CONTROL "&LIVE", DC_LIVE, "button", BS_AUTOCHECKBOX | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 296, 61, 45, 12, 0
CONTROL "Submission", -1, "static", SS_LEFT | WS_CHILD | WS_VISIBLE, 6, 137, 42, 8, 0
CONTROL "", -1, "static", SS_ETCHEDFRAME | WS_CHILD | WS_VISIBLE, 48, 142, 292, 1, 0
CONTROL "&Bulk Submission File:", -1, "static", SS_LEFT | WS_CHILD | WS_VISIBLE, 12, 150, 72, 8, 0
CONTROL "", DC_BULK, "edit", ES_AUTOHSCROLL | WS_CHILD | WS_VISIBLE | WS_BORDER | WS_TABSTOP, 88, 148, 200, 12
CONTROL "Browse...", DC_BROWSE, "button", BS_PUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 292, 148, 45, 12, 0
CONTROL "", -1, "static", SS_ETCHEDFRAME | WS_CHILD | WS_VISIBLE, 6, 166, 335, 1, 0
CONTROL "&Create", IDOK, "BUTTON", BS_DEFPUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 232, 172, 50, 14
CONTROL "Cancel", IDCANCEL, "BUTTON", BS_PUSHBUTTON | BS_CENTER | WS_CHILD | WS_VISIBLE | WS_TABSTOP, 287, 172, 50, 14
}
#endresource
 Steven Horowitz has been working for Novaworks for over five years as a technical expert with a focus on EDGAR HTML and XBRL. Since the creation of the Legato language in 2015, Steven has been developing scripts to improve the GoFiler user experience. He is currently working toward a Bachelor of Sciences in Software Engineering at RIT and MCC. |
Additional Resources
Legato Script Developers LinkedIn Group
Primer: An Introduction to Legato
Quicksearch
Categories
Calendar
Feeds
