Package 'ScatterDensity'

Title: Density Estimation and Visualization of 2D Scatter Plots
Description: The user has the option to utilize the two-dimensional density estimation techniques called smoothed density published by Eilers and Goeman (2004) <doi:10.1093/bioinformatics/btg454>, and pareto density which was evaluated for univariate data by Thrun, Gehlert and Ultsch, 2020 <doi:10.1371/journal.pone.0238835>. Moreover, it provides visualizations of the density estimation in the form of two-dimensional scatter plots in which the points are color-coded based on increasing density. Colors are defined by the one-dimensional clustering technique called 1D distribution cluster algorithm (DDCAL) published by Lux and Rinderle-Ma (2023) <doi:10.1007/s00357-022-09428-6>.
Authors: Michael Thrun [aut, cre, cph] (ORCID: <https://orcid.org/0000-0001-9542-5543>), Felix Pape [aut, rev], Luca Brinkman [aut], Quirin Stier [aut] (ORCID: <https://orcid.org/0000-0002-7896-4737>)
Maintainer: Michael Thrun <[email protected]>
License: GPL-3
Version: 0.1.4
Built: 2026-10-21 18:13:40 UTC
Source: https://github.com/mthrun/scatterdensity

Help Index


Density Estimation and Visualization of 2D Scatter Plots

Description

The user has the option to utilize the two-dimensional density estimation techniques called smoothed density published by Eilers and Goeman (2004) <doi:10.1093/bioinformatics/btg454>, and pareto density which was evaluated for univariate data by Thrun, Gehlert and Ultsch, 2020 <doi:10.1371/journal.pone.0238835>. Moreover, it provides visualizations of the density estimation in the form of two-dimensional scatter plots in which the points are color-coded based on increasing density. Colors are defined by the one-dimensional clustering technique called 1D distribution cluster algorithm (DDCAL) published by Lux and Rinderle-Ma (2023) <doi:10.1007/s00357-022-09428-6>.

Details

The DESCRIPTION file:

Package: ScatterDensity
Type: Package
Title: Density Estimation and Visualization of 2D Scatter Plots
Version: 0.1.4
Date: 2026-08-25
Authors@R: c(person("Michael", "Thrun", email= "[email protected]",role=c("aut","cre","cph"), comment = c(ORCID = "0000-0001-9542-5543")),person("Felix", "Pape",role=c("aut","rev")),person("Luca","Brinkman",role=c("aut")),person("Quirin", "Stier",role=c("aut"), comment = c(ORCID = "0000-0002-7896-4737")))
Maintainer: Michael Thrun <[email protected]>
Description: The user has the option to utilize the two-dimensional density estimation techniques called smoothed density published by Eilers and Goeman (2004) <doi:10.1093/bioinformatics/btg454>, and pareto density which was evaluated for univariate data by Thrun, Gehlert and Ultsch, 2020 <doi:10.1371/journal.pone.0238835>. Moreover, it provides visualizations of the density estimation in the form of two-dimensional scatter plots in which the points are color-coded based on increasing density. Colors are defined by the one-dimensional clustering technique called 1D distribution cluster algorithm (DDCAL) published by Lux and Rinderle-Ma (2023) <doi:10.1007/s00357-022-09428-6>.
LazyLoad: yes
Imports: Rcpp, RcppParallel (>= 5.1.4), pracma
Suggests: DataVisualizations, ggplot2, ggExtra, plotly, FCPS, parallelDist, secr, ClusterR, geometry, sf
Depends: methods, R (>= 2.10)
LinkingTo: Rcpp, RcppArmadillo, RcppParallel
NeedsCompilation: yes
License: GPL-3
Encoding: UTF-8
URL: https://www.deepbionics.org/
BugReports: https://github.com/Mthrun/ScatterDensity/issues
Config/pak/sysreqs: cmake
Repository: https://mthrun.r-universe.dev
Date/Publication: 2026-10-21 13:00:52 UTC
RemoteUrl: https://github.com/mthrun/scatterdensity
RemoteRef: HEAD
RemoteSha: 4e16f1bf8c23c375231378bf133badb1565f2498
Author: Michael Thrun [aut, cre, cph] (ORCID: <https://orcid.org/0000-0001-9542-5543>), Felix Pape [aut, rev], Luca Brinkman [aut], Quirin Stier [aut] (ORCID: <https://orcid.org/0000-0002-7896-4737>)

Index of help topics:

Cluster2Polygon         Construct a Convex or Concave Boundary for One
                        Cluster
DDCAL                   Density Distribution Cluster Algorithm of [Lux
                        and Rinderle-Ma, 2023].
DensityScatter.DDCAL    Scatter density plot [Brinkmann et al., 2023]
fast_table_num          fast_table_num
inPSphere2D             2D data points in Pareto Sphere
InteractiveGate         Interactive manual gating of scatter pplots
InteractiveSequentialGates
                        Interactive Sequential Gates
Mask2Polygon            Construct a Convex Geometry from a Logical
                        Population Mask
Mutualinfo_pde_2d       Mutual Information from a Two-Dimensional
                        Pareto Density Estimate
ParetoDensityEstimation2D
                        Paired Two-Dimensional Pareto Density Extension
ParetoDensityEstimationFromFit
                        Re-evaluate a Pareto Density After Omitting
                        Observations
PDEscatter              Scatter Density Plot
PointsInPolygon         PointsInPolygon
PolygonGate             PolygonGate
SampleScatter           takes a sample for a scatter plot
ScatterDensity-package
                        Density Estimation and Visualization of 2D
                        Scatter Plots
SmoothedDensitiesXY     Smoothed Densities X with Y

Author(s)

Michael Thrun [aut, cre, cph] (ORCID: <https://orcid.org/0000-0001-9542-5543>), Felix Pape [aut, rev], Luca Brinkman [aut], Quirin Stier [aut] (ORCID: <https://orcid.org/0000-0002-7896-4737>)

Maintainer: Michael Thrun <[email protected]>

Examples

#Todo

Construct a Convex or Concave Boundary for One Cluster

Description

Constructs a two-dimensional geometric boundary for one selected cluster using the first two columns of Data.

With Ratio = 1, the function returns the convex hull. With Ratio < 1, it requests a concave hull using sf::st_concave_hull() when the selected cluster contains at least three non-collinear hull points.

Usage

Cluster2Polygon(
  Data,
  Cls,
  ClusterNo,
  Ratio = 1,
  PlotIt = FALSE
)

Arguments

Data

Matrix or data frame containing observations in rows and variables in columns. At least two columns are required. Only the first two columns are used for geometry construction, and both must be numeric.

Cls

Vector of cluster assignments. Its length must equal nrow(Data). Missing cluster labels are ignored when selecting the requested cluster.

ClusterNo

Scalar cluster identifier specifying which observations are used. Comparison with Cls is performed after conversion to character.

Ratio

Numeric scalar in [0,1] controlling the requested geometry. Ratio = 1 returns the convex hull; Ratio < 1 requests a concave hull using sf::st_concave_hull(). Smaller values generally allow a more concave boundary. Default is 1.

PlotIt

Logical scalar. If TRUE, the finite unique observations belonging to the selected cluster and the resulting geometry are plotted with base R graphics. Default is FALSE.

Details

Observations are selected using

!is.na(Cls) & as.character(Cls) == as.character(ClusterNo).

Only the first two columns of Data are used. Rows containing NA, NaN, Inf, or -Inf in either coordinate are removed before constructing the geometry. Duplicate coordinate pairs are also removed.

A convex hull is always determined first and is used to detect degenerate clusters:

  • one hull point gives "POINT";

  • two hull points give "LINESTRING";

  • three or more hull points give "POLYGON".

For one or two unique finite observations, the observations themselves define the hull. Otherwise hull indices are calculated with grDevices::chull().

If Ratio = 1, the convex geometry is returned directly. Convex polygon paths are explicitly closed by appending the first vertex.

If Ratio < 1 and at least three non-collinear hull points are available, package sf is required. The selected coordinates are converted to a MULTIPOINT geometry and passed to

sf::st_concave_hull(..., ratio = Ratio, allow_holes = FALSE).

If Ratio < 1 but the cluster is degenerate, a concave polygon is not defined. The corresponding POINT or LINESTRING is returned instead and a warning is issued. In this case sf is not required.

When PlotIt = TRUE, selected observations are plotted and the resulting geometry is overlaid in red.

Value

A list containing exactly four elements:

Vertices

Numeric matrix with columns VertexNo, X, and Y. Each row is one vertex of the returned geometry. Convex polygon output is explicitly closed by repeating the first polygon vertex at the end.

ClusterNo

The cluster identifier supplied to the function.

Ratio

The ratio used to construct the geometry.

GeometryType

Character scalar describing the geometry returned for the selected cluster. Possible values include "POINT", "LINESTRING", and "POLYGON".

Note

The function validates that Data has at least two columns, length(Cls) == nrow(Data), ClusterNo contains exactly one non-missing value, Ratio is finite and lies in [0,1], and the first two columns used for geometry construction are numeric.

Author(s)

Michael Thrun

See Also

chull

Examples

Data <- data.frame(
  X = c(0, 1, 1, 0, 3, 4, 4, 3),
  Y = c(0, 0, 1, 1, 0, 0, 1, 1)
)

Cls <- c(1, 1, 1, 1, 2, 2, 2, 2)

poly <- Cluster2Polygon(
  Data = Data,
  Cls = Cls,
  ClusterNo = 1,
  Ratio = 1
)

poly$Vertices
poly$GeometryType

## Not run: 
concave <- Cluster2Polygon(
  Data = Data,
  Cls = Cls,
  ClusterNo = 1,
  Ratio = 0.5,
  PlotIt = TRUE
)

## End(Not run)

Density Distribution Cluster Algorithm of [Lux and Rinderle-Ma, 2023].

Description

DDCAL is a clustering-algorithm for one-dimensional data, which heuristically finds clusters to evenly distribute the data points in low variance clusters.

Usage

DDCAL(data, nClusters, minBoundary = 0.1, maxBoundary = 0.45,

numSimulations = 20, csTolerance = 0.45, csToleranceIncrease = 0.5)

Arguments

data

[1:n] Numeric vector, with the data values

nClusters

Scalar, number of clusters to be found

minBoundary

Scalar, in the range (0,1), gives the lower boundary (in percent), for the simulation. Default is 0.1

maxBoundary

Scalar, in the range (0,1), gives the upper boundary (in percent), for the simulation. Default is 0.45

numSimulations

Scalar, number of simulations/iterations of the algorithm

csTolerance

Scalar, in the range (0,1). Gives cluster size tolerance factor. The necessary cluster size is defined by (dataSize/nClusters - dataSize/nClusters * csTolerance). Default is 0.45

csToleranceIncrease

Scalar, in the range (0,1), gives the procentual increase of the csTolerance-factor, if some clusters did not reach the necessary size. Default is 0.5

Details

DDCAL creates a evenly spaced division of the min-max-normalized data from minBoundary to maxBoundary. Those divisions will be used as boundaries. The first initial clusters will be the data from min(data) to minBoundary and maxBoundary to max(data). The clusters will be extended to neighboring points, as long as the standard deviations of the clusters will be reduced. A potential clusters will be used, if they have the necessary size, given as (dataSize/nClusters - dataSize/nClusters * csTolerance). If both clusters can be used, the left cluster (which is the cluster from min(data) to minBoundary or above) is preferred. If no clusters can be found with the necessary size, then the csTolerance-factor and with it the necessary cluster size will be lowered. If a clusters is used, the next boundaries are found, which are not in the already existing clusters and the procedure is repeated with the not already clustered data, until all points are assigned to clusters.

If a matrix is given as input data, the first column of the matrix will be used as data for the clustering

Non-finite values will not be clustered, but instead will get the cluster label NaN.

The algorithm is not garantueed to produce the given number of clusters, given in nClusters. The found number of clusters can be lower, depending on the data and input parameters.

Value

labels

[1:n] Numeric vector, containing the labels for the input data points

Author(s)

Luca Brinkmann

References

[Lux and Rinderle-Ma, 2023] Lux, M., Rinderle-Ma, S.: DDCAL: Evenly Distributing Data into Low Variance Clusters Based on Iterative Feature Scaling; Springer Journal of Classification, Vol. 40, pp. 106-144, DOI: doi:10.1007/s00357-022-09428-6, 2023.

Examples

# Load data
if(requireNamespace("FCPS")){
data(EngyTime, package = "FCPS")
engyTimeData = EngyTime$Data
c1 = engyTimeData[,1]
c2 = engyTimeData[,2]
}else{
c1 = rnorm(n=4000)
c2 = rnorm(n=4000,1,2)
}
# Calculate Densities

densities = SmoothedDensitiesXY(c1,c2)$Densities
# Use DDCAL to cluster the densities
labels = DDCAL(densities, 9)

# Plot Densities according to labels
my_colors = c("#000066", "#3333CC", "#9999FF", "#00FFFF", "#66FF33",
                       "#FFFF00", "#FF9900", "#FF0000", "#990000")
labels = as.factor(labels)
df = data.frame(c1, c2, labels)

if(requireNamespace("ggplot2")){
ggplot2::ggplot(df, ggplot2::aes(c1, c2, color = labels)) +
  ggplot2::geom_point() +
  ggplot2::scale_color_manual(values = my_colors)
}

Scatter density plot [Brinkmann et al., 2023]

Description

Density estimation (PDE) [Ultsch, 2005] or "SDH" [Eilers/Goeman, 2004] used for a scatter density plot, with clustering of densities with DDCAL [Lux/Rinderle-Ma, 2023] proposed by [Brinkmann et al., 2023].

Usage

DensityScatter.DDCAL(X, Y, nClusters = 12, Plotter = "native", 
DensityMethod = "SDH", LimitShownPoints = FALSE,
Marginals = FALSE, na.rm=TRUE, pch, Size, 
xlab="x", ylab="y", main = "",lwd = 2,
xlim=NULL,ylim=NULL,Polygon,BW = TRUE,Silent = FALSE, ...)

Arguments

X

Numeric vector [1:n], first feature (for x axis values)

Y

Numeric vector [1:n], second feature (for y axis values)

nClusters

(Optional) Integer defining the number of clusters (colors) used for finding a hard color transition, default is 12.

Plotter

(Optional) String, name of the plotting backend to use. Possible values are: "native", "plotly", or "ggplot2"

DensityMethod

(Optional) character, "SDH", "PDEsphere" or "PDEjoint"

LimitShownPoints

(Optional) FALSE: does nothing, TRUE: samples the number of optimal points for visualization using SampleScatter

Marginals

(Optional) Boolean, if TRUE the marginal distributions of X and Y will be plotted together with the 2D density of X and Y. Default is FALSE

na.rm

(Optional) Boolean, if TRUE non finite values will be removed

pch

(Optional) Scalar or character. Indicates the shape of data points, see plot function, symbol argument in plotly package, or the shape argument in ggplot2 package, default is 20 for native and for ggplot2, and 0 for plotly

Size

(Optional) Scalar, size of data points in plot, default is 1 for native, 6 for plotly, and 3 for ggplot2

xlab

(Optional) String, title of the x axis. Default: "X", see plot() function, or similar functonality in plotly or ggplot2

ylab

(Optional) String, title of the y axis. Default: "Y", see plot() function, or similar functonality in plotly or ggplot2

main

(Optional) Character, title of the plot.

lwd

(Optional) Scalar, thickness of the lines used for the marginal distributions (only needed if Marginals=TRUE), see plot(). Default = 2

xlim

(Optional) numerical vector, min and max of x values to be plottet

ylim

(Optional) numerical vector, min and max of y values to be plottet

Polygon

(Optional) [1:p,1:2] numeric matrix that defines for x and y coordinates a polygon in magenta

BW

(Optional) Boolean, if TRUE and Plotter="ggplot2" will use a white background, if FALSE and Plotter="ggplot2", the typical ggplot2 background is used. Not needed if "Plotter="native". Default is TRUE

Silent

(Optional) Boolean, if TRUE no messages will be printed, default is FALSE

...

Further plot arguments

Details

The DensityScatter.DDCAL function generates the density of the xy data as a z coordinate. Afterwards xyz will be plotted as a contour plot. It assumens that the cases of x and y are mapped to each other meaning that a cbind(x,y) operation is allowed. The colors for the densities in the contour plot are calculated with DDCAL, which produces clusters to evenly distribute the densities in low variance clusters.

In the case of "native" as Plotter, the handle returns NULL because the basic R functon plot() is used.

For the returned density values see SmoothedDensitiesXY or PDEscatter depending on input parameter SDHorPDE for details.

Value

returns a invisible list with

DF

[1:m,1:5] of Density values, x values, y values, colors, and classification vector Cls. m=n if LimitShownPoints=FALSE, otherwise LimitShownPoints=TRUE m<n meaning that subsample is taken

PlotHandle

the plotting handle, either an object of plotly, ggplot2 or NULL depending on input parameter Plotter

Author(s)

Luca Brinkmann, Michael Thrun

References

[Ultsch, 2005] Ultsch, A.: Pareto density estimation: A density estimation for knowledge discovery, In Baier, D. & Werrnecke, K. D. (Eds.), Innovations in classification, data science, and information systems, (Vol. 27, pp. 91-100), Berlin, Germany, Springer, 2005.

[Eilers/Goeman, 2004] Eilers, P. H., & Goeman, J. J.: Enhancing scatterplots with smoothed densities, Bioinformatics, Vol. 20(5), pp. 623-628. 2004.

[Lux/Rinderle-Ma, 2023] Lux, M. & Rinderle-Ma, S.: DDCAL: Evenly Distributing Data into Low Variance Clusters Based on Iterative Feature Scaling, Journal of Classification vol. 40, pp. 106-144, 2023.

[Brinkmann et al., 2023] Brinkmann, L., Stier, Q., & Thrun, M. C.: Computing Sensitive Color Transitions for the Identification of Two-Dimensional Structures, Proc. Data Science, Statistics & Visualisation (DSSV) and the European Conference on Data Analysis (ECDA), p.109, Antwerp, Belgium, July 5-7, 2023.

Examples

# Create two bimodial distributions
x1=rnorm(n = 7500,mean = 0,sd = 1)
y1=rnorm(n = 7500,mean = 0,sd = 1)
x2=rnorm(n = 7500,mean = 2.5,sd = 1)
y2=rnorm(n = 7500,mean = 2.5,sd = 1)
x=c(x1,x2)
y=c(y1,y2)

DensityScatter.DDCAL(x, y, Marginals = TRUE)

fast_table_num

Description

Faster table computation in R compared to table() by omitting as.character and as.factor

Usage

fast_table_num(x, y, edges_x, edges_y, redefine = TRUE,

byrow = FALSE, all.inside = FALSE, rightmost.closed = FALSE,

sort = FALSE, na.rm = FALSE, names = FALSE,extendOutput=FALSE)

Arguments

x

[1:n] numerical vector

y

[1:n] numerical vector

edges_x

Optional, [1:(k1k_1+1)] numerical vector defining the specific borders in x default unique(x) for categorical scale

edges_y

Optional, [1:(k2k_2+1)] numerical vector defining the specific borders in y, default unique(y) for categorical scale

redefine

Optional, boolean TRUE: resets counts in y direction in order from 1:k2k_2 to k2k_2:1

byrow

Optional, boolean, If FALSE (the default) the count matrix is filled by columns, otherwise the matrix is filled by rows.

all.inside

Optional, boolean, if TRUE, the returned indices are coerced into 1,...,N-1, i.e., 0 is mapped to 1 and N to N-1

rightmost.closed

Optional, boolean, if TRUE, the rightmost interval, vec[N-1] .. vec[N] is treated as closed

sort

Optional, boolean, if TRUE, edges_x, edges_y are sorted non-decreasingly, NA/NaN are gthen ignored

na.rm

Optional, boolean, if TRUE, only complete observations are taken into account

names

Optional, boolean, if TRUE, output matrix is named by edges_x[1:k1k_1] and edges_y[1:k2k_2] (left-sided)

extendOutput

Optional, boolean, default FALSE, if TRUE, list is the output, otherwise numerical matrix, see below.

Details

edges_x and edges_y must be sorted non-decreasingly. Beware that kernels are centers of bins, edges_x, edges_y are borders of bins. If edges are given, edges_x, edges_y can contain Inf,-Inf borders. In that case, edges always define n-1 bins lying within the edges. data outside first edge or last edge are ignored. Edges have either to be sorted non-decreasingly or set sort=T.

If edges are not given, set sort=T. In this case, they define the unique number of points. Then the number of edges internally sets the number of bins.

Beware that in matrix notation, count matrix would be expected be ordered [1:k1k_1,1:k2k_2] instead of [1:k2k_2,1:k1k_1]. Here we use the ordering that intuitively is given in plot(x,y), i.e. x are columns and y are rows.

Value

extendOutput==FALSE: [1:k2k_2,1:k1k_1] numerical matrix of counts

If extendOutput==TRUE, then list of named elements

count matrix: [1:k2k_2,1:k1k_1] numerical matrix of counts

x_idx: [1:k1k_1] numerical vector of counts for x based on edges_x

y_idx: [1:k2k_2] numerical vector of counts for y based on edges_y

Author(s)

Michael Thrun

See Also

table, findInterval

Examples

if(requireNamespace("FCPS")){
data(Hepta,package ="FCPS")
Cls=Hepta$Cls
Cls1=Cls+1
#k unqiue points define k bins
fast_table_num(Cls,Cls1,

redefine = FALSE,names=TRUE)==as.matrix(table(Cls,Cls1))
}
#k unqiue points define k bins
tab=fast_table_num(rnorm(100),rnorm(100),redefine=FALSE,sort=TRUE)

#set k+1 edges to get k bins
x=rnorm(100)
y=rnorm(100)
binsxy=5
edgex=seq(from=min(x),to=max(x),length.out=binsxy+1)
edgesy=seq(from=min(y),to=max(y),length.out=binsxy+1)
fast_table_num(x,y,edgex,edgesy,

redefine=FALSE,names=TRUE,rightmost.closed =TRUE)

#definition of counts analog to plotting
x = c(rnorm(1000, mean=-5), rnorm(1000, mean=5))
y = rnorm(2000)
edgesx = seq(min(x), max(x), length.out=512+1)
edgesy = seq(min(y), max(y), length.out=256+1)
joint_table = fast_table_num(x, y, edgesx, edgesy)

plot(x,y)
plot(colSums(joint_table),xlab="x marginal",

ylab="sum of counts",main="x-values are stored in columns")

plot(rowSums(joint_table),xlab="y marginal",

ylab="sum of counts",main="y-values are stored in rows")

2D data points in Pareto Sphere

Description

This function determines the 2D data points inside a ParetoSphere with ParetoRadius.

Usage

inPSphere2D(data, paretoRadius=NULL,Compute="Cpp")

Arguments

data

numeric matrix of data.

paretoRadius

numeric value. radius of P-spheres. If not given, calculate by the function 'paretoRad'

Compute

Either Cpp or Parallel, defines if multicore computation should be performed

Value

numeric vector with the number of data points inside a P-sphere with ParetoRadius.

Author(s)

Felix Pape


Interactive manual gating of scatter pplots

Description

Allows to draw a polygon around a specifc area in a two-dimensional scatter plot

Usage

InteractiveGate(Data, GateVars, Gatename = "Leukocytes", PlotIt = FALSE)

Arguments

Data

[1:n,1:d] numerical matrix of n cases and d variables. each variable has a name accessible via colnames(Data)

GateVars

[1:2] character vector of columns that are selected

Gatename

name of the gate

PlotIt

TRUE: plots the result

Details

Only polygon gates can be used in the interactive tool.

Value

a LIST,

Polygon

[1:p,1:2] numerical matrix of coordinates for variables named byGateVars defining the polygon

Index

indices of data points within the polygon

DataInGate

[1:m,1:d] numerical matrix of n cases and d variables of data points within the polygon

Author(s)

Michael Thrun

See Also

gs_gate_interactive

Examples

Disk=""
#path=ReDi("AutoGating/09Originale",Disk)
#V=ReadLRN("110001_T1_NB_d11_N33k",path)
#Data=V$Data
#V=InteractiveGate(Data,GateVars = c( "SS", "FS"),PlotIt = T)

Interactive Sequential Gates

Description

Performs several consecutive gates interactively on the same data. Only polygon gates can be used in the interactive tool.

Usage

InteractiveSequentialGates(Data, GateVarsMat, Gatenames, PlotIt = FALSE)

Arguments

Data

[1:n,1:d] numerical matrix of n cases and d variables. each variable has a name accessible via colnames(Data)

GateVarsMat

[1:2,1:c] character matrix of columns that are selected, each rwo represents one gate

Gatenames

[1:c] character vector of names of the sequential gates

PlotIt

TRUE: plots the last gate on the prior of last data

Details

In flow cytometry, the process of setting sequential or consecutive gates is known as interactive gating. Interactive gating involves analyzing and selecting specific populations of cells or particles based on their characteristics in a step-by-step manner.

Once the initial gate is set, you can further refine the analysis by examining the population within that gate. By assessing the patterns and characteristics of the cells within the initial gate, you can determine which parameters are relevant for subsequent gating.

Based on the analysis of the first gate, you can then create a second gate or subsequent gates to isolate more specific subsets of cells. This process involves creating new scatter plots based on the relevant parameters identified earlier. The successive gates are usually set based on the characteristics observed within the previously gated populations.

Value

a LIST,

Polygons

[1:c] list named by Gatenames, each element is a [1:p,1:2] numerical matrix of coordinates for variables named byGateVars defining the polygon

Indices

[1:c] list named by Gatenames, each element defines the indices of data points within the polygon with regards to the data of the prior polygon (gate)

DataInGates

[1:c] list named by Gatenames, each element is a [1:m,1:d] numerical matrix of n cases and d variables of data points within the current polygon

Author(s)

Michael Thrun

See Also

gs_gate_interactive

Examples

Disk=""
#path=ReDi("AutoGating/09Originale",Disk)
#read data
#filename="110001_T1_NB_d11_N33k"
#V=dbt.DataIO::ReadLRN(filename,path)
#Data=V$Data
#find out names of given parameters
#colnames(Data)
#select parameters, each two define one gate
#GateVarsMat=rbind(c("SS", "FS"),c("CD45", "CD19"))
#name the gates, the first one is overall data, every next one is a consecutive gate
#base on the gated information of the previous gate
#Gatenames=c("Lymphocytes","Leukocytes")
#apply interactive function
#GateInfo=ScatterDensity::InteractiveSequentialGates(Data,
#GateVarsMat=GateVarsMat,Gatenames =Gatenames#,PlotIt = T)

Construct a Convex Geometry from a Logical Population Mask

Description

Constructs a convex two-dimensional geometry for observations selected by a logical mask.

The mask defines membership in a population, cluster, or hierarchy node, while GateVars defines the two variables used for its geometric representation.

Usage

Mask2Polygon(
  Data,
  Mask,
  ClusterNo,
  GateVars,
  PlotIt = FALSE
)

Arguments

Data

Matrix or data frame containing observations in rows and variables in columns.

Mask

Logical vector of length nrow(Data) defining which observations belong to the population.

TRUE selects an observation and FALSE excludes it. Missing values in Mask are treated as FALSE.

ClusterNo

Scalar identifier of the population, cluster, or hierarchy node.

The value is passed to Cluster2Polygon and included in the resulting geometry representation.

GateVars

Vector defining the two columns of Data used as X and Y coordinates.

Elements may be column names or column indices. At least two variables must be supplied. If more than two variables are supplied, only the first two are used and a warning is issued.

The two selected variables must be different.

PlotIt

Logical. If TRUE, the observations selected by Mask and the resulting convex geometry are plotted by Cluster2Polygon. Defaults to FALSE.

Details

Population membership is determined exclusively by Mask. For example, a hierarchical child population may be defined as:

ChildMask <- ParentMask & LocalGateRule

In contrast, GateVars only determines the local two-dimensional coordinate system in which the selected population is represented.

Missing values in Mask are replaced by FALSE. If the resulting mask does not select any observations, the function stops with an error.

The selected rows and the two gate variables are extracted from Data. A local cluster vector is then constructed by assigning ClusterNo to every selected observation.

The resulting data are passed to:

Cluster2Polygon(
  Data = GateData,
  Cls = LocalCls,
  ClusterNo = ClusterNo,
  Ratio = 1,
  PlotIt = PlotIt
)

Consequently, Mask2Polygon always constructs a convex hull or, for a degenerate set of selected observations, the corresponding point or line geometry.

The geometric hull and the original logical mask are not necessarily equivalent. In particular, observations that were not selected by Mask may geometrically lie inside the convex hull of the selected observations.

The returned geometry can be used as input for downstream geometric gating functions such as PointsInPolygon or PolygonGate.

Value

List V with following elements

Polygon a numerical matrix/data.frame with X and Y, ClusterNo, Ratio- An open polygon is closed internally.

Author(s)

Michael Thrun, IAP-GmbH Intelligent Analytics Projects

See Also

Cluster2Polygon, PointsInPolygon, PolygonGate

Examples

Data <- matrix(
  c(
    0, 0,
    1, 0,
    1, 1,
    0, 1,
    5, 5
  ),
  ncol = 2,
  byrow = TRUE
)

colnames(Data) <- c("X", "Y")

Mask <- c(TRUE, TRUE, TRUE, TRUE, FALSE)

Polygon <- Mask2Polygon(
  Data = Data,
  Mask = Mask,
  ClusterNo = 1,
  GateVars = c("X", "Y")
)

Polygon

## Not run: 
# Optional visual inspection
Mask2Polygon(
  Data = Data,
  Mask = Mask,
  ClusterNo = 1,
  GateVars = c("X", "Y"),
  PlotIt = TRUE
)

## End(Not run)

Mutual Information from a Two-Dimensional Pareto Density Estimate

Description

Estimates mutual information between two continuous variables from the joint density returned by ParetoDensityEstimation2D.

Usage

Mutualinfo_pde_2d(x, y, base = 2)

Arguments

x

Numeric vector containing observations of the first continuous variable.

y

Numeric vector containing observations of the second continuous variable. It must satisfy the input requirements of ParetoDensityEstimation2D.

base

Numeric scalar defining the logarithm base. It must be finite, positive, and different from 1. Default is 2, yielding mutual information in bits.

Details

The function first estimates the joint density using ParetoDensityEstimation2D.

The density grid is converted to a probability-mass matrix using trapezoidal quadrature:

pij=f(xi,yj)wi(X)wj(Y).p_{ij} = f(x_i,y_j) w_i^{(X)} w_j^{(Y)}.

The mass matrix is normalized to sum to one. Marginal probability masses are then calculated from its row and column sums.

Mutual information is evaluated as:

I(X;Y)=∑i,jpijlog⁡b(pijpi⋅p⋅j),I(X;Y) = \sum_{i,j} p_{ij} \log_b\left( \frac{p_{ij}}{p_{i\cdot}p_{\cdot j}} \right),

where bb is the logarithm base supplied through base. Only cells with positive joint probability mass are included.

A small negative result caused by floating-point error is replaced by zero.

Value

Numeric scalar containing the estimated mutual information.

With the default base = 2, the result is expressed in bits. With base = exp(1), it is expressed in nats.

Note

The estimator inherits all sample-size, uniqueness, and dependency requirements of ParetoDensityEstimation2D.

In particular, at least 50 paired observations and at least 12 unique values per variable are required.

See Also

ParetoDensityEstimation2D

Examples

## Not run: 
set.seed(1)

x <- rnorm(200)
y <- x + rnorm(200, sd = 0.5)

Mutualinfo_pde_2d(x, y)

# Result in natural-logarithm units
Mutualinfo_pde_2d(x, y, base = exp(1))

## End(Not run)

Paired Two-Dimensional Pareto Density Extension

Description

Constructs a two-dimensional density estimate for two paired continuous variables. The estimator combines published one-dimensional Pareto Density Estimates with a paired rectangular joint-density seed and iterative proportional fitting.

The resulting joint distribution preserves the observed pairing between X and Y while matching the two one-dimensional Pareto Density Estimate marginals.

Usage

ParetoDensityEstimation2D(X, Y)

Arguments

X

Numeric vector containing observations of the first continuous variable.

Y

Numeric vector containing observations of the second continuous variable. Must have the same length as X.

Details

The function performs the following steps:

  1. Estimate the one-dimensional Pareto densities of X and Y using DataVisualizations::ParetoDensityEstimation().

  2. Extract the Pareto radii, kernel locations, and density estimates returned by the one-dimensional estimator.

  3. If a one-dimensional kernel grid contains more than 256 locations, interpolate the published density to a grid of 256 locations. This limits quadratic memory growth and is not an additional smoothing operation.

  4. Convert the one-dimensional density estimates to normalized probability masses using trapezoidal quadrature.

  5. Construct a paired rectangular joint-density seed:

    q(s,t)=∑iI{∣Xi−s∣≤rX}I{∣Yi−t∣≤rY}.q(s,t) = \sum_i I\{|X_i-s| \leq r_X\} I\{|Y_i-t| \leq r_Y\}.

  6. Convert the joint seed to probability mass and add a very small positive support floor.

  7. Apply iterative proportional fitting so that the row and column marginals agree with the two one-dimensional Pareto Density Estimate probability masses.

  8. Convert the fitted probability mass back to a density and calculate density values at the observed pairs using bilinear interpolation.

The IPF algorithm stops when the maximum marginal error is less than 1e-8, or after 500 iterations. A warning is issued if the tolerance is not reached.

This function is an explicit paired rectangular two-dimensional extension constrained by published one-dimensional Pareto Density Estimates. It is not claimed to be a published two-dimensional hypersphere Pareto Density Estimator.

Value

A list with components:

Densities

Numeric vector containing the bilinearly interpolated joint-density value at each observed pair (X[i], Y[i]).

GridDensity

Numeric matrix containing the estimated joint density on the rectangular grid. Rows correspond to Xkernels; columns correspond to Ykernels.

Xkernels

Numeric vector containing the kernel-grid locations for X.

Ykernels

Numeric vector containing the kernel-grid locations for Y.

Note

At least 50 paired observations and at least 12 unique values in each variable are required.

All observations must be finite. Remove missing, infinite, or non-finite paired observations before calling this function.

The suggested package DataVisualizations is required.

See Also

Mutualinfo_pde_2d, ParetoDensityEstimation

Examples

## Not run: 
set.seed(1)

X <- rnorm(200)
Y <- 0.7 * X + rnorm(200, sd = 0.6)

fit <- ParetoDensityEstimation2D(X, Y)

length(fit$Densities)
dim(fit$GridDensity)
length(fit$Xkernels)
length(fit$Ykernels)

## End(Not run)

Re-evaluate a Pareto Density After Omitting Observations

Description

Re-evaluates a previously fitted one-dimensional Pareto density after omitting one or more observations. The original Pareto radius, full-sample boundaries, counting grid, interpolation decision, and interpolation target remain fixed.

Usage

ParetoDensityEstimationFromFit(V, Omit = NULL)

Arguments

V

Result returned by ParetoDensityEstimation(). It must contain kernels, paretoRadius, and a non-NULL Fit component. The fit context must contain Data, paretoCounts, and countingKernels.

Omit

Optional numeric vector of finite integer indices referring to observations in V$Fit$Data. Repeated indices are ignored. NULL or an empty vector performs no omission.

Details

The function supports leave-one-out or leave-several-out re-evaluation without re-estimating the structural settings of the original Pareto density.

If V$Fit$countingKernels is NULL, V$kernels are used as the counting grid. Otherwise, the stored counting kernels are used.

For each omitted observation, one count is removed from every Pareto sphere whose interval contains that observation. If the observation lies within one Pareto radius of the lower or upper full-sample boundary, the corresponding reflected contribution is removed as well.

The updated counts are normalized by trapezoidal integration over the fixed evaluation grid. If the original fit used interpolation, the normalized density is interpolated back to V$kernels with stats::approx(). Non-finite interpolated values are replaced by zero.

For compatibility with older fit objects that do not store interpolationUsed, interpolation is inferred from whether countingKernels is non-NULL.

Value

A list with components:

kernels

Numeric vector of evaluation locations, unchanged from V$kernels.

paretoDensity

Numeric vector of Pareto-density values after removing the requested observations and renormalizing the remaining counts.

paretoRadius

Unchanged Pareto radius from V.

Note

A numerical Dirac representation has no usable fit context and cannot be re-evaluated by this function.

See Also

ParetoDensityEstimation

Examples

## Not run: 
fit <- DataVisualizations::ParetoDensityEstimation(
  Data = rnorm(200),
  PlotIt = FALSE
)

fit_minus_one <- ParetoDensityEstimationFromFit(fit, Omit = 1)
fit_minus_many <- ParetoDensityEstimationFromFit(fit, Omit = c(1, 2, 2, 10))

## End(Not run)

Scatter Density Plot

Description

Concept of Pareto density estimation (PDE) proposed for univsariate data by [Ultsch, 2005] and comparet to varius density estimation techniques by [Thrun et al., 2020] for univariate data is here applied for a scatter density plot. It was also applied in [Thrun and Ultsch, 2018] to bivariate data, but is not yet compared to other techniques.

Usage

PDEscatter(x,y,SampleSize,

na.rm=FALSE,PlotIt=TRUE,ParetoRadius,Compute="Cpp", 

sampleParetoRadius, NrOfContourLines=20,Plotter='native',

DrawTopView = TRUE, xlab="X", ylab="Y", main="PDEscatter",
                              
xlim, ylim, Legendlab_ggplot="value", NormalizeAsDensity=FALSE)

Arguments

x

Numeric vector [1:n], first feature (for x axis values)

y

Numeric vector [1:n], second feature (for y axis values)

SampleSize

Numeric m, positiv scalar, maximum size of the sample used for calculation. High values increase runtime significantly. The default is that no sample is drawn

na.rm

Function may not work with non finite values. If these cases should be automatically removed, set parameter TRUE

ParetoRadius

Numeric, positiv scalar, the Pareto Radius. If omitted (or 0), calculate by paretoRad.

Compute

Either Cpp or Parallel, defines if multicore computation should be performed

sampleParetoRadius

Numeric, positiv scalar, maximum size of the sample used for estimation of "kernel", should be significantly lower than SampleSize because requires distance computations which is memory expensive

PlotIt

TRUE: plots with function call

FALSE: Does not plot, plotting can be done using the list element Handle

-1: Computes density only, does not perfom any preperation for plotting meaning that Handle=NULL

NrOfContourLines

Numeric, number of contour lines to be drawn. 20 by default.

Plotter

String, name of the plotting backend to use. Possible values are: "native", "ggplot", "plotly"

DrawTopView

Boolean, True means contur is drawn, otherwise a 3D plot is drawn. Default: TRUE

xlab

String, title of the x axis. Default: "X", see plot() function

ylab

String, title of the y axis. Default: "Y", see plot() function

main

string, the same as "main" in plot() function

xlim

see plot() function

ylim

see plot() function

Legendlab_ggplot

String, in case of Plotter="ggplot" label for the legend. Default: "value"

NormalizeAsDensity

Boolean, TRUE: normalizes densities to integral equal to one

Details

The PDEscatter function generates the density of the xy data as a z coordinate. Afterwards xyz will be plotted either as a contour plot or a 3d plot. It assumens that the cases of x and y are mapped to each other meaning that a cbind(x,y) operation is allowed. This function plots the PDE on top of a scatterplot. Variances of x and y should not differ by extreme numbers, otherwise calculate the percentiles on both first. If DrawTopView=FALSE only the plotly option is currently available. If another option is chosen, the method switches automatically there.

The method was succesfully used in [Thrun, 2018; Thrun/Ultsch 2018].

PlotIt=FALSE is usefull if one likes to perform adjustements like axis scaling prior to plotting with ggplot2 or plotly. In the case of "native"" the handle returns NULL because the basic R functon plot() is used

Value

List of:

X

Numeric vector [1:m],m<=n, first feature used in the plot or the kernels used

Y

Numeric vector [1:m],m<=n, second feature used in the plot or the kernels used

Densities

Numeric vector [1:m],m<=n, Number of points within the ParetoRadius of each point, i.e. density information

Matrix3D

1:n,1:3] marix of x,y and density information

ParetoRadius

ParetoRadius used for PDEscatter

Handle

Handle of the plot object. Information-string if native R plot is used.

Note

MT contributed with several adjustments

Author(s)

Felix Pape

References

[Thrun/Ultsch, 2018] Thrun, M. C., & Ultsch, A. : Effects of the payout system of income taxes to municipalities in Germany, in Papiez, M. & Smiech,, S. (eds.), Proc. 12th Professor Aleksander Zelias International Conference on Modelling and Forecasting of Socio-Economic Phenomena, pp. 533-542, Cracow: Foundation of the Cracow University of Economics, Cracow, Poland, 2018.

[Ultsch, 2005] Ultsch, A.: Pareto density estimation: A density estimation for knowledge discovery, In Baier, D. & Werrnecke, K. D. (Eds.), Innovations in classification, data science, and information systems, (Vol. 27, pp. 91-100), Berlin, Germany, Springer, 2005.

[Thrun et al., 2020] Thrun, M. C., Gehlert, T. & Ultsch, A.: Analyzing the Fine Structure of Distributions, PLoS ONE, Vol. 15(10), pp. 1-66, DOI doi:10.1371/journal.pone.0238835, 2020.

Examples

#taken from [Thrun/Ultsch, 2018]
if(requireNamespace("DataVisualizations")){
data("ITS",package = "DataVisualizations")
data("MTY",package = "DataVisualizations")
Inds=which(ITS<900&MTY<8000)
plot(ITS[Inds],MTY[Inds],main='Bimodality is not visible in normal scatter plot')


PDEscatter(ITS[Inds],MTY[Inds],xlab = 'ITS in EUR',

ylab ='MTY in EUR' ,main='Pareto Density Estimation indicates Bimodality' )


}

PointsInPolygon

Description

Defines a Cls based on points in a given polygon.

Usage

PointsInPolygon(Points, Polygon, PlotIt = FALSE, ...)

Arguments

Points

[1:n,1:2] xy cartesian coordinates of a projection

Polygon

Polygon a numerical matrix/data.frame with X and Y, optionally VertexNo, or list containing Vertices or X and Y. VertexNo, ClusterNo, Ratio, and GeometryType are optional; if VertexNo is supplied, it is used to restore the path order. An open polygon is closed internally.

PlotIt

TRUE: Plots marked points

...

BMUorProjected: Default == FALSE, If TRUE assuming BestMatches of ESOM instead of Projected Points

main: title of plot

Further Plotting Arguments,xlab etc used in Classplot

Details

We assume that polygon is closed, i.e., that the last point connects to the fist point

Value

Numerical classification vector Cls with 1 = outside polygon and 2 = inside polygon

Author(s)

Michael Thrun

See Also

Classplot

Examples

XY=cbind(runif(80,min = -1,max = 1),rnorm(80))
#closed polygon
polymat <- cbind(x = c(0,1,1,0), y = c(0,0,1,1))
#takes sometimes more than 5 sec

Cls=PointsInPolygon(XY,polymat,PlotIt = TRUE)

PolygonGate

Description

A specific Gate defined by xy coordinates that result in a closed polygon is applied to the flowcytometry data.

Usage

PolygonGate(Data, Polygon, GateVars,  PlotIt = FALSE, PlotSampleSize = 1000)

Arguments

Data

numerical matrix n x d

Polygon

Polygon a numerical matrix/data.frame with X and Y, optionally VertexNo, or list containing Vertices or X and Y. VertexNo, ClusterNo, Ratio, and GeometryType are optional; if VertexNo is supplied, it is used to restore the path order. An open polygon is closed internally.

GateVars

vector, either column index in Data of X and Y coordinates of gate or its variable names as string

PlotIt

if TRUE: plots a sample of data in the two selected variables and marks point inside the gate as yellow and outside as magenta

PlotSampleSize

size pof the plottet sample

Details

Gates are alwaxs two dimensional, i.e., require two filters, although all dimensions of data are filted by the gates. Only high-dimensional points inside the polygon (gate) are given back

Value

list of

DataInGate

m x d numerical matrix with m<=n of data points inside the gate

InGateInd

index of length m for the datapoints in original matrix

Note

if GateVars is not found a text is given back which will state this issue

Author(s)

Michael Thrun

See Also

PointsInPolygon

Examples

Data <- matrix(runif(1000), ncol = 10)
colnames(Data)=paste0("GateVar",1:ncol(Data))
poly <- cbind(x = c(0.2,0.5,0.8), y = c(0.2,0.8,0.2))
#set PlotIt TRUE for understanding the example


#Select index
V=PolygonGate(Data,poly,c(5,8),PlotIt=FALSE,100)

#select var name
V=PolygonGate(Data,poly,c("GateVar5","GateVar8"),PlotIt=FALSE,100)

takes a sample for a scatter plot

Description

Given 2D points having X and Y coordinates takes a sample, such that these points are is optimally visualized if a plot function is called.

Usage

SampleScatter(X, Y, ThresholdPoints = 20,

DensityThreshold, nbins = 100,na.rm=TRUE, PlotIt = FALSE)

Arguments

X

[1:n] x coordinate

Y

[1:n] y coordinate

ThresholdPoints

how many points per grid are allowed to overlap, default 20

DensityThreshold

default is internally estimated, scalar above zero and below 1, SDH threshold for density computed with SmoothedDensitiesXY, usually good values are around 0.1

nbins

number of bins in grid, default 100x100

na.rm

Function may not work with non finite values. If these cases should be automatically removed, set parameter TRUE

PlotIt

Plots the remaining points

Details

"Optimally"" visualized in the sense that not too much point overap visually. The lower the value for ThresholdPoints, the smaller is the sample that is taken by the function.

Value

SubsampleInd[1:m] indices of m points, m<n, that will be relevant for a optimal scatter plot

Author(s)

Michael Thrun

See Also

SmoothedDensitiesXY

Examples

if(requireNamespace("DataVisualizations")){
data("ITS",package = "DataVisualizations")
data("MTY",package = "DataVisualizations")
sample_ind=SampleScatter(ITS,MTY,PlotIt=TRUE)
}else{
#sample random data
ITS=rnorm(10000)
MTY=rnorm(10000)
sample_ind=SampleScatter(ITS,MTY,ThresholdPoints = 5)
del_ind=setdiff(1:length(ITS),sample_ind)
plot(ITS,MTY,type="p",pch=20,col="grey",main="Grey=full data, red=overlapping data points")
points(ITS[del_ind],MTY[del_ind],type="p",pch=20,col="red")
}

Smoothed Densities X with Y

Description

Density is the smothed histogram density at [X,Y] of [Eilers/Goeman, 2004]

Usage

SmoothedDensitiesXY(X, Y, nbins, lambda, Xkernels, Ykernels, 

isXDiscrete=FALSE, isYDiscrete=FALSE, Compute="Cpp", PlotIt = FALSE)

Arguments

X

Numeric vector [1:n], first feature (for x axis values)

Y

Numeric vector [1:n], second feature (for y axis values),

nbins

Optional, number of bins, default is 250, if scalar nbins=nx=ny otherwise c(nx,ny) vector of length 2

lambda

Optional, smoothing factor used by the density estimator, default is lambda = 20

Xkernels

Optional, bin kernels in x direction are given, if not set nx bins in range of data are generated with seq

Ykernels

Optional, bin kernels y direction are given, if not set ny bins in range of data are generated with seq

isXDiscrete

whether or not the first numeric vector resembles a continuous or discrete measurement

isYDiscrete

whether or not the second numeric vector resembles a continuous or discrete measurement

Compute

Either R, Cpp or Parallel, defines how the heavy-load computation is performed

PlotIt

FALSE: no plotting, TRUE: simple plot

Details

lambda has to chosen by the user and is a sensitive parameter and a lambda = 20 roughly means that the smoothing is over 20 bins around a given point.

Value

List of:

Densities

numeric vector [1:n] is the smothed density in 3D

Xkernels

numeric vector [1:nx], nx defined by nbins, such that mesh(Xkernels,Ykernels,F) form the ( not NaN) smothed densisties

Ykernels

numeric vector [1:ny], nx defined by nbins, such that mesh(Xkernels,Ykernels,F) form the ( not NaN) smothed densisties

GridDensity

matrix [1:nx,1:ny] beeing the smoothed 2D histogram

Points2GridInd

an index such that Densities = GridDensity[Points2GridInd]

Author(s)

Michael Thrun

References

[Eilers/Goeman, 2004] Eilers, P. H., & Goeman, J. J.: Enhancing scatterplots with smoothed densities, Bioinformatics, Vol. 20(5), pp. 623-628.DOI: doi:10.1093/bioinformatics/btg454, 2004.

Examples

if(requireNamespace("DataVisualizations")){
data("ITS",package = "DataVisualizations")
data("MTY",package = "DataVisualizations")
Inds=which(ITS<900&MTY<8000)
V=SmoothedDensitiesXY(ITS[Inds],MTY[Inds])
}else{
#sample random data
ITS=rnorm(1000)
MTY=rnorm(1000)
V=SmoothedDensitiesXY(ITS,MTY)
}