1. Dr. Spotfire: Scaling Spotfire
Peter McKinnis
Architect, Analytics Enterprise Architect Group
December 3, 2019

4. © Copyright 2000-2019 TIBCO Software Inc.
About TIBCO
TIBCO fuels digital business by enabling better decisions and faster, smarter actions through the TIBCO Connected Intelligence Cloud. From APIs and systems to devices and
people, we interconnect everything, capture data in real time wherever it is, and augment the intelligence of
your business through analytical insights. Thousands
of customers around the globe rely on us to build
compelling experiences, energize operations, and
propel innovation. Learn how TIBCO makes
digital smarter at 
© Copyright TIBCO Software Inc.

5. © Copyright 2000-2019 TIBCO Software Inc.
Architecture review
Scaling considerations
Spotfire Server
Web Player
Helpful links
© Copyright TIBCO Software Inc.

6. What to get out of presentation?
I hope you…
Learn something new
Get some pointers to resources that will be helpful
Get some strategies and information to help you size and scale your environments
What you will not get
Some secret magic formula for computing the exact scale and size of an environment
Go to Monitoring & Diagnostics and Action Logs and Performance Counter Example
© Copyright TIBCO Software Inc.

7. TIBCO Spotfire Architecture
Spotfire Analyst
Consumer and Business Author
Spotfire Server
Web Player
Mobile Access
© Copyright TIBCO Software Inc.

8. Scaling Considerations - Users
Types of users
Number of concurrent users of each type
Or Total number of users and can estimate concurrent (concurrent is typically 10-20% of total)
Business Author
Analyst
Consumer
Spotfire Server
Web Player
Spotfire Server
© Copyright TIBCO Software Inc.

9. Scaling Considerations - Data Size
Amount of RAM used by Spotfire analysis with data loaded
Estimated Number of rows and columns of data
Estimated database size of datasets
Relative size can be useful. For example:
2X-Large
> 200 GB
X-Large GB
Large
10’s GB
Medium 1-10 GB
Small
100MB-1GB
X-Small
< 100 MB
© Copyright TIBCO Software Inc.

10. Scaling Considerations - Data Consumption
In-Memory
In-Memory
Load data from source into memory.
External Data
Leave data in database. Dynamically load and discard data to visualize.
External
On-demand
On-Demand
Dynamically swap data in and out of memory.
Streaming Data
Streaming Data
Real-time data loaded into memory and out of memory as time window shifts.
© Copyright TIBCO Software Inc.

11. Scaling Considerations – In-memory
If already have Spotfire analysis available, one can estimate RAM using Spotfire Analyst
Open Spotfire Analyst and look at memory used by Spotfire.Dxp.exe in Task Manager:
Open Spotfire Analysis in Spotfire Analyst and look at memory used by Spotfire.Dxp.exe in Task Manager:
Consider formula of data size + user memory (typically MB – depending on complexity of analysis)
This file - 1 Million rows x 100 Cols, about 70% character and 30% numeric columns - uses about 1.9 GB = GB GB
Simple formula: Size_of_data + (user_data * # users) = MemoryNeeded – user_data typically between 40-80MB depending on analysis complexity.
For example: (60 MB * 100) = 7.9 GB
© Copyright TIBCO Software Inc.

12. Scaling Considerations – Data in-memory
# of Rows
# of Columns
# of Cells
(Rows * Columns)
# of Numeric Columns
# of Character Columns
Size in-memory (GB)
Table size in SQL Server (GB)
1,000,000
100
30,000,000 30 70
1.95
1.6
1,500,000 9
13,500,000 2 7
0.6
0.190
10,000,000 40
400,000,000 25 14
5.9
4.0
15,000,000
135,000,000
7.5
1.9
50,000,000 20
1,000,000,000 10
13.9
13.1
In-memory data size affected by:
entropy of data (a few or many unique values in a column)
data types of columns (character, numeric, number of characters)

13. Scaling Considerations – Spotfire Analyses
Number of Analyses
What is largest analysis in-memory?
Complexity of Analyses:
Calculations - calculated columns, custom expressions
Scripts - IronPython, TERR, etc.
Can affect CPU load
Is analysis pre-loaded by Scheduled Updates?
This saves a lot of CPU by not having to load the file for each user.
Recommended for highly used analyses and important analyses that take a while to load
Is data shared by users?
Reduces memory footprint in Web Player
How to setup caching for reports with row-level security using personalized Information Links
© Copyright TIBCO Software Inc.

14. © Copyright 2000-2019 TIBCO Software Inc.
When to scale?
New environment:
Use information regarding – users, data, etc. to scale environment
Consider - How precise do I need to be in my sizing? Can I start small and scale up as needed or do I need to know how big for the next 3 years?
Changes in environment:
New Spotfire Analyses being deployed
Increase in Spotfire users
Increase in amount of data in-memory
Increase in number of on-demand queries
Need to support high availability
Addition of Streaming data
Performance issues in environment:
Need to determine where the performance issue might be (could be external issue)
Is hardware at capacity – memory, CPU, disk?
Current view in Monitoring & Diagnostics Administration Web UI
Historical information available from:
Action Logs and System Monitoring
Performance Counter Log Files
Go to Monitoring & Diagnostics and Action Logs and Performance Counter Example
© Copyright TIBCO Software Inc.

15. TIBCO Spotfire Server - Scaling
Items most related to scaling Spotfire Server:
Number of concurrent data connections via Information Services (on-demand and other queries)
Number of concurrent active Consumers/Business Authors and, to some extent, Analyst users
Need for high availability
Streaming data used on Web Player (10.6 and above) – increases amount of HTTP traffic since visuals constantly updating
What to scale?
Scale up - Additional memory and CPUs
Increase JVM Memory for Spotfire Server
In Spotfire Server configuration, Increase max-jobs and thread-pool-size (Information Services jobs)
Scale out - Additional Spotfire Servers
Use same Spotfire database so install server and configure to use existing database
Need load balancer with sticky sessions enabled in front of 2 or more Spotfire Servers
Spotfire Servers tend to not be too large, e.g. 4-8 cores and GB RAM
Show how to increase JVM RAM on 10.3.x
© Copyright TIBCO Software Inc.

16. TIBCO Spotfire Web Player Service – Scaling
Items most related to scaling Web Player Service:
Number of concurrent active Web Player users
Largest Analysis file needed in-memory
Complexity of calculations in analysis
Sharing of data in analysis
Use of Scheduled Updates
How data will be consumed:
In-memory
External
Data on-demand
Streaming added in 10.6 for Web Player (See Scaling streaming visual analytics with Spotfire Web Player)
What to scale?
Scale up
Additional memory and CPUs
If scale up very large, additional Web Player instances on the same node provides more throughput.
Scale out - Additional Web Player Nodes
Web Player Nodes tend to larger than Spotfire Server, e.g cores and GB RAM
© Copyright TIBCO Software Inc.

17. © Copyright 2000-2019 TIBCO Software Inc.
Helpful Links
Web Player Performance Information:
Accessing performance data of Services
Web Player service performance counters
Troubleshooting Performance issues on TIBCO Spotfire Web Player
Streaming data - Spotfire 10.6 and later - Scaling streaming visual analytics with Spotfire Web Player
Spotfire Server Configuration:
Spotfire Server hangs or responds slowly with error: “Too many jobs”
Spotfire Server JVM Memory Settings:
Modifying the virtual memory 10.3.x
TIBCO Spotfire Server in a cluster crashes due to long GC pauses
© Copyright TIBCO Software Inc.

18. © Copyright 2000-2019 TIBCO Software Inc.