Large tenants and many operations can produce large audit logs in Delegate365 since every operation and every sync jobs is protocolled. To optimize the logging and to offer export and further use of the operations data, the audit logging changes with this version. See the details here.
- Previous versions: Let's have a look back into the logging in the past. In the early versions of Delegate365, the logging was done in a database which blew up the database (unnecessarily). So, with version 2, we changed the logging to simple text files in CSV format to make it very easy to work with the protocolled data, for example, in Microsoft Excel. This format is used up to Delegate365 version 5.x.
- Room for improvement: For the logging in the existing form, we were struggling with three difficulties:
First, if an admin wants to work with the protocolled data, the logfiles need to be downloaded manually from the Delegate365 portal (as ZIP file including multiple CSV files or as XLSX file with one day in one sheet).
Secondly, the log shows the changed data in a long text string (separated with comma, as in the sample below).
One log entry looks as here:
22,11/29/2016 11:04:53 AM,User,admin@CIE123815.onmicrosoft.com,mollyc@CIE123815.onmicrosoft.com,
Modified,Seattle,"License: none; Field: City, OldValue: , NewValue: Seattle;
Field: PhysicalDeliveryOfficeName, OldValue: , NewValue: Seattle; Field: StateOrProvince, OldValue: , NewValue: Washington"
Thirdly, the web space is limited. So, if large logs are produced, the website cannot store the logfiles forever, but just maybe for a year. We wanted to improve that. - New Logging (overview): As explained above, we wanted to optimize the logging. So, we completely redesigned the logging system in Delegate365. The following graphics gives an overview how the logging in version 6 works: The "old" logging still is in place and will be removed in near future. The "new" logging inserts any operation into a queue. An own job reads the new messages of the logging queue and processes the messages. There's a switch that protocols the message into the Delegate365 database and additionally in an Azure storage.
So, from now on, there is "short-term logging" and "long-term logging" (colored green in the the graphics above). - Old logging: Until version 5.x, logging is done only into the web storage of the Delegate365 instance. This space is limited (usually about 10GB). This logging is still in place additionally, to make the old logs available for some time. We plan to remove the old logging completely in one of the next versions.
- Logging in the database: In the database, the log entries will be existing up to 3 months (90 days). The idea is to have the last log data available in the UI very quickly and searchable within the Delegate365 portal. Usually, if any actions concerning specific users need to be found, the admin gets an information from his colleagues within days. So he can look into the last operations - if they happened within the last 90 days - within the portal quickly. The logging system takes care about the cleanup and removes all older entries automatically.
- Logging in a storage: One goal was to have (more or less) unlimited storage for logs, so that Portal Admins no longer have to take care of downloading logfiles themselves, if the limited space runs out. The second goal was to provide an easy access for Portal Admins to get the logs without the need for manual downloading them from the Delegate365 portal. So we introduced Azure Table Storage into the Delegate365 ecosystem. Every Delegate365 instance gets an own Azure storage for their logs. This happens automatically with the upgrade to version 6.
- New logging format: Instead of values separated with commas we now use own columns and JSON format. We will post the new format here shortly.
- Direct Access of Azure storage: Besides the theoretically unlimited space in Azure storage, data in that storage can be accessed directly (if you have permissions). As tool for accessing your Delegate365 logfiles we recommend to download the cost free Microsoft Azure Storage Explorer (see also here).
This allows Portal Admins to connect to an Azure Storage to query and to download the produced log files. The key will be available in the Delegate365 portal. - Support for Power-BI: With the new logging, you can connect directly to the Delegate365 Audit Logs and use the data for your own queries. We will show guidelines or a ready to use dashboard in the next weeks. This was one of our goals to make Power BI available for all operations data.
- Removing reports: In return, we will remove the reporting section in Delegate365 in one of the next versions. Admins shall benefit from pivoting their Delegate365 logs as needed. We think this is the better approach to work with the data in other systems as Microsoft Excel or Power-BI.
- How To use logging data and Power-BI: There will be a step-by-step blogpost here shortly showing the procedure of exporting data and using data from Delegate365 with Power-BI.
We are currently still doing the final touches on version 6. This update will take some more days, but we are working intensively on the new version and testing. So, stay tuned, more will follow!