A generalized net model of the stochastic gradient descent and dropout algorithm with intuitionistic fuzzy evaluations

: In the paper, we consider a stochastic gradient descent algorithm in combination with a dropout method. We used the theory of intuitionistic fuzzy sets for the assessment of the equivalence of the respective assessment units. We also consider a degree of uncertainty when the information is not enough.


Introduction
Stochastic gradient descent is a very popular and common algorithm used in various machine learning algorithms, the most important being the basis of Neural Networks (NN). Gradient descent is a method of finding a local extremum (minimum or maximum) of a function by moving along the gradient. Dropout [25] works by switching off neurons in a network during training to force the remaining neurons to take on the load of the missing neurons. This is typically done randomly with a certain percentage of neurons per layer being switched off. To find the average weight of each neuron, we use avgk and avgk is the average weight input of a neuron on the k-th layer and Wjk (i) is the matrix of the weight for the current iteration i before beginning the training and n is the number of neurons in the k-th layer.
In the present work, we use the apparatus of intuitionistic fuzzy sets, defined by Atanassov [1,2] in 1983 as an extension of the theory of fuzzy sets created by L. Zadeh [28].
Let Е be a fixed set. The set А* is called intuitionistic fuzzy set if there is: where functions µA : Е → [0; 1] and νA : Е → [0; 1], set respectively the degree of membership and non-membership of the elements x ∈ E to the set А, which is a subset of Е and for each x ∈ E: The function πА that sets the degree of uncertainty of the membership of the elements x ∈ E to the set А is determined by the formula: In the case of a fuzzy set πА(х) = 0, for each x ∈ E. The comparison between the elements of any two Intuitionistic fuzzy sets, say A and B, involves a double comparison between the degree of membership and non-membership of the respective elements to the two networks.
In intuitionistic fuzzy logic (IFL) [4,6], the degree of membership and non-membership can be noted as: where m is the lower boundary of the "narrow" range; u -the upper boundary of the "broad" range; n -the upper boundary of the "narrow" range.

Generalized nets
Generalized nets (GNs) [3,5,7] are defined in a way that is principally different from the ways of defining the other types of Petri nets. During the time GN have become a tool for modelling parallel operating systems. Models for neural networks [8,9] and data mining methods [11−14] have been developed. The first basic difference between GNs and ordinary Petri nets is the "place -transition" relation. Here the transitions are objects of a more complex nature. A transition may contain m input places and n output places where m, n ≥ 1.
Formally, every transition is described by a seven-tuple ( (c) t2 is the current value of the duration of its active state; (d) r is the condition of the transition to determine which tokens will pass (or transfer) from the inputs to the outputs of the transition; it has the form of an Index Matrix: ri,j is the predicate that corresponds to the i-th input and j-th output place. When its truth value is "true", a token from the i-th input place transfers to the j-th output place; otherwise, this is not possible; (e) M is an IM of capacities of transition's arcs:  The following tokens stay in the generalized net.

Generalized net model
• In place SG -one αG token with characteristic "Random number generator" for generalizing weight coefficients.
• In each place SF one αi token, 1 ≤ i ≤ k, with the characteristic "Transfer of a function from the i-th layer to the neural network".
• In place ST -one αt token with characteristic "Learning objective for neural network output". • In place SEZ -one αez token with characteristic "Pre-fixed error in neural network training". The generalized net includes the following set of seven transitions: where the following events take place: • Z3 -calculating the gradient; • Z4 -calculating the outputs аk = FK(nk) from the k-th layer; • Z5 -determining the difference between the received value (SO) and the fixed learning target and the least-square error between them; • Z6 -determining whether the artificial neural network (ANN) has been learnt or not; • Z7 -calculating the new weight coefficients. Each of the seven transitions is described below in detail.
Transition Z1 has the following form: -number of neurons whose value is greater than the average value for the layer -m; -number of neurons whose value is less than the average value for the layer -f. Initially, we calculate the average value for the layer, We obtain , in case when > , we obtain the degree of membership having the following form: We obtain +, , in case when < , we obtain the degree of non-membership having the following form: We obtain /0 " , in case when = , we obtain the uncertainty: The following new values can be obtained:

Conclusions
A new generalized net model, simulation of the neural network learning process combining the Dropout Method and Stochastic Gradient Descent are considered. The model makes it possible to consider the different stages in the training of the neural network. An estimation with intuitionistic fuzzy sets is used. The intuitionistic fuzzy evaluations reflect the results of the system. A degree of uncertainty is also considered in case of insufficient information. A generalized net model is used to describe the whole process.